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SUMMARY 


Dynamic  Help,  an  online  documentation  system  that 
displays  a  context-specific  and  state-specific  message,  has 
been  given  a  prototype  implementation  in  the  US  Army 
software  package  ACIFS  (Automated  Central  Issue  Facility 
System) .  A  Dynamic  Help  message,  assembled  or  retrieved  tor 
the  specific  screen,  cursor  location  and  data  state,  tries 
to  answer  all  relevant  questions  (Where  am  I?  What  can  I  do 
here?  .  .  . ) •  This  thesis  reports  a  user  test  of  the 

prototype,  performed  as  part  of  the  Embedded  User  Support 
project  at  Georgia  Tech  for  the  US  Army  Installation  Support 
Modules  (ISM)  project  office. 

A  group  of  26  US  Army  supply  clerks  typifying  the 
intended  population  of  novice  and  intermittent  users  of 
ACIFS  perfoinmed  two  versions  (one  with  Dynamic  Help 
available,  one  without)  of  challenging  tasks.  The  tasks 
were  designed  to  be  realistic  and  dense  in  challenges 
(situations  where  novice  and  intermittent  users  might  not  be 
able  to  proceed  without  some  sort  of  aid) . 

The  primary  aim  of  the  user  testing  was  to  determine  if 
Dynamic  Help  would  improve  ACIFS 's  accuracy,  usability, 
productivity,  as  measured  respectively  by  differences  in 
error  counts,  differences  in  Dead  End  counts  (where  a  Dead 
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End  is  that  point  in  a  task  where  the  user  requires  outside 
assistance  to  proceed) ,  differences  in  task  durations,  and 
user-satisfaction  responses  (measured  by  a  direct  question 
with  a  5-point  response  from  1  =  Very  Helpful  to  5  =  Very 
Unhelpful) .  Supplementing  this  evaluative  aim  was  a 
diagnostic  one:  to  study  how  and  under  what  conditions 
Dynamic  Help  could  influence  performance  and  satisfaction. 

The  ACIFS  program  was  modified  to  provide  automatic 
collection  of  all  durations,  certain  errors,  and  diagnostic 
indicators  such  as  counts  and  times  of  important  keystrokes. 
Two  of  the  26  test  clerks  were  used  to  run  a  pilot  test,  and 
their  data  was  discarded.  The  author  administered  testing 
to  the  remaining  clerks  in  daily  pairs,  each  pair  performing 
four  tasks  in  the  morning  and  four  in  the  afternoon.  Usable 
performance  data  on  all  tasks  was  collected  from  22  users, 
who  each  performed  four  pairs  of  tasks  with  and  without 
Dynamic  Help,  in  randomized  order.  For  some  measures  for 
some  users  for  some  tasks,  portions  of  the  automatically 
collected  data  were  lost  or  not  collected,  or  users  failed 
to  complete  some  parts  of  the  task. 

Users  accessed  Dynamic  Help  on  average  4.682  times  per 
task,  spending  an  average  of  46.623  seconds  with  each  help 
message  (from  access  to  field  entry) . 

Dynamic  Help  significantly  improved  usability,  not  only 
as  a  whole  but  for  every  task.  Users  averaged  a  total  of 
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5.682  Dead  Ends  without  Dynamic  Help  and  only  2.091  Dead 
Ends  with  Dynamic  Help. 

In  error  performance,  the  users'  behavior  split  the 
data  into  two  groups:  the  nine  Complete-Error-Data  users 
for  whom  complete  error  data  on  all  tasks  was  available,  and 
all  other  users.  For  Complete-Error-Data  users.  Dynamic 
Help  reduced  the  four-task  number  of  errors  per  user  from 
7.11  to  5.555  (significant  in  a  paired  t-test  at  the  0.035 
level).  The  other  users  committed  more  errors  (8.73  errors 
per  user  for  four  tasks) ,  and  committed  more  (but  not 
significantly  more)  errors  with  Dynamic  Help  than  without. 

In  time  performance.  Dynamic  Help  did  not  save  time 
either  for  Complete-Error-Data  users  or  for  others,  but 
neither  did  it  significantly  slow  users  down.  Under  test 
conditions,  user  carelessness,  lack  of  time  discipline,  and 
the  Dead  End  alternative  to  spending  time  degraded  the 
experiment's  ability  to  measure  Dynamic  Help's  effect  on 
performance  time.  Dynamic  Help  does  not  appear  to  have 
potential  for  saving  time  unless  the  messages  can  be 
restructured  to  highlight  and  bring  to  the  top  the  sentences 
most  likely  to  be  helpful. 

Users  seemed  satisfied  with  Dynamic  Help's  helpfulness. 
18  users  classified  it  as  "Very  Helpful";  the  other  6  as 
"Somewhat  Helpful",  Responses  did  indicate  that  Dynamic 
Help  would  perform  better  with  restructuring. 
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The  author  also  found  that  Dynamic  Help  decreased  the 
number  of  wrong  guesses  for  certain  data  entry  locations  in 
the  software.  Dynamic  Help  was  not  helpful  in  menu 
navigation,  and  is  not  useful  when  users  do  not  realize  the 
presence  of  challenges  to  accuracy  and  efficiency. 

Recommendations  include  (1)  a  restructuring  of  both 
menu  selection  and  data-entry  screen  help  messages  and  (2) 
additional  assistance  in  data-entry  screens  including  a 
separate  key  to  access  lists  of  available  choices.  Also, 
future  user  tests  should  be  designed  to  more  closely 
resemble  an  actual  job  setting  in  terms  of  accuracy  and  time 
discipline. 


CHAPTER  I 


COMPUTER  INTERFACE  IMPROVEMENT 

There  are  many  aspects  of  usability  that  the  designer 
must  consider  during  the  development  of  a  computer  system, 
such  as  system  performance,  user  interface,  system 
functions,  installation  and  field  maintenance  (Gould,  1988) . 
Among  these,  the  user  interface  is  an  area  which  has 
recently  gained  increasing  recognition,  and  it  is  the  only 
one  that  can  be  economically  improved  after  the  fact, 
because  a  face  is  easier  to  change  than  a  fundamental 
structure . 

The  design  of  a  computer  system  whose  interface  is 
transparent  to  any  targeted  user  is  a  complex  problem  that 
is  well  documented.  Many  computer  systems  are  developed  for 
a  particular  type  of  user,  such  as  a  novice,  intermittent  or 
expert  user.  However,  regardless  of  the  user  type,  a  high 
quality  user  interface  will  strive  to  include  consistent 
command  languages,  clear  operation  sequences,  organized 
display  formats,  consistent  terminology,  complete 
instructions,  simple  error  recovery  procedures  and  clear, 
nonthreatening  error  messages  (Shneiderman,  1987) , 
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The  needs  of  an  expert  user  are  different  from  that  of 
the  novice  user  or  the  intermittent  user  (Shneiderman, 

1987)  .  The  frequent  user  is  familiar  with  the  syntactic  as 
well  as  the  semantic  aspects  of  the  computer  system  while 
the  novice  user  has  no  knowledge  of  syntax  and  may  only  have 
very  little  semantic  knowledge  of  the  computer  system.  The 
intermittent  user  will  have  knowledge  of  the  system,  but 
because  he  only  uses  it  occasionally,  his  syntactical 
knowledge  may  be  very  weak.  Because  of  these  differences 
between  user  types,  their  user  interface  requirements  will 
be  quite  different.  The  novice  user  will  be  most  concerned 
with  avoiding  errors,  while  the  expert  user  will  be 
concerned  with  speed  of  task  completion.  The  novice  user 
requires  a  system  with  a  small  set  of  easy-to-learn  commands 
and  informative  feedback  after  the  accomplishment  of  each 
action.  The  expert  user  is  concerned  with  getting  the  most 
work  done  through  the  least  number  of  keystrokes  or  actions 
and  desires  minimum  feedback.  The  needs  of  the  intermittent 
user  fall  somewhere  between  those  of  the  expert  and  the 
novice.  Usually,  intermittent  users  will  desire  a  system 
which  allows  them  to  move  through  the  program  with  the  least 
demands  on  memory.  Clear  menus  and/or  prompts  assist  them 
in  recognizing  the  proper  sequence  of  actions  to  perform. 
Also,  they  desire  meaningful  feedback  to  assure  them  that 
tasks  are  properly  completed. 
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Designing  a  computer  system  for  any  one  of  these  user 
types  i  .s  cult.  Designing  one  which  meets  the  needs  of 

more  than  one  type  is  especially  difficult. 

1.1  Interface  Difficulties  For  Novice  or  Intermittent  Users 
Novice  and  intermittent  users  of  a  computer  system  are 
not  as  confident  as  the  expert  user  in  their  ability  to 
interact  with  the  computer.  They  are  generally  more 
cautious  and  slower  in  responding  to  the  system,  and  hence, 
their  productivity  is  lower  than  that  of  the  freguent  or 
expert  user.  An  expert  user  or  a  user  who  specializes  in  a 
single  function  and  operates  only  one  or  two  software 
modules  can  tolerate  and  overcome  great  deficiencies  in  user 
interface  design  and  user  support.  The  user's 
familiarization  and  skills  are  refreshed  daily  and  is 
amortized  over  thousands  of  performances.  But,  novice  or 
intermittent  users  will  not  enjoy  this  continual  familiarity 
with  a  software  package.  Instead,  each  time  they  use  a 
package,  they  will  have  to  relearn  some  of  the  skills, 
techniques,  and  commands  that  are  required  to  accomplish  a 
task.  The  problems  of  novice  and  intermittent  users  are 
compounded  when  software  packages  do  not  share  reasonably 
consistent  and  easy-to-learn  interfaces  (Young,  1990a) . 
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1.2  Approaches  to  Solving  Interface  Design  Problems 
There  are  three  general  ways  to  improve  an  existing 
computer  system's  user  interface:  fundamental  redesign, 
interface  redesign,  and  embedded  user  support  (Young, 

1990a)  . 

1.2.1  Fundamental  Redesign 

First,  the  entire  computer  program  could  be  rewritten 
from  start  to  finish,  integrating  consistency  in 
linguistics,  style,  and  interface  across  the  different 
functions  within  the  package.  Many  interface  difficulties 
arise  from  deficiencies  in  data  structures  and  functional 
design.  When  the  fundamental  cause  is  awkwardly  defined 
quantities  poorly  designed  procedures,  a  complete  rewrite 
can  eliminate  interface  problems  or  sidestep  them.  If  the 
goal  is  to  improve  the  user  interfaces  of  several  different 
types  of  software,  each  computer  program  in  the  group  could 
be  rewritten,  with  the  incorporation  of  a  consistent  set  of 
linguistics,  style  and  interfaces  across  all  the  entire 
group.  This  would  provide  a  common  platform  and  interface 
to  all  packages,  allowing  a  user  of  any  package  to  quickly 
learn  the  interface  of  another  package  in  the  group.  The 
chief  drawback  to  any  rewrite  policy  is  expense.  The 
rewriting  of  multiple  software  packages  to  conform  to  a 
common  user  interface  design  may  take  many  man-years  of 
effort. 
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1.2.2  Interface  Redesign 


Thfc  alternative  in  improving  a  deficient  user 

interface  is  to  rewrite  only  user  interfaces.  Programming 
effort  can  be  reduced  by  patching  a  shell  over  the  existing 
interface,  but  this  may  result  in  the  program's  being  slowed 
down.  The  shell  would  be  another  layer  of  computer  code  to 
process . 

1.2.3  Embedded  User  Support 

In  some  cases,  it  takes  too  much  time  and  money  to 
rewrite  a  complete  set  of  software  packages  or  even  to 
rewrite  the  user  interface.  Besides  the  programming  time 
and  expense,  rewriting  either  whole  packages  or  interfaces 
creates  a  need  for  retraining  of  users.  What  is  desired  in 
such  cases  is  an  economically  attractive  "patch"  that  with 
minimum  time  and  effort  can  improve  the  usability  of  a 
computer  program  or  set  of  programs  by  helping  users  to  get 
past  interface  deficiencies.  Thus,  a  third  alternative  in 
improving  the  user  interface  is  with  Embedded  User  Support 
(EUS)^.  Embedded  User  Support  involves  adding  dynamic  help 
and  documentation  and/or  embedded  tutorials  to  an  existing 
computer  program.  It  does  not  attempt  to  change  or  improve 


^EUS ,  pronounced  "use"  as  in  "useful,"  is  an  acronym 
for  Embedded  User  Support  and  is  also  the  first  part  of  the 
word  "eusophy,"  which  means  "having  the  quality  of  being 
easy  to  learn,"  from  the  Greek  "eu-"  (easy  or  well)  and 
"sophy"  (wisdom  or  skill) . 
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upon  the  existing  user  interface.  Instead,  it  guides  the 
user  past  pitfalls  and  shortcomings  in  the  interface. 

Another  benefit  of  EUS  is  that  users  with  different  skill 
levels  can  use  the  same  interface  without  a  degradation  in 
performance.  All  users  will  have  Help  available  whether 
they  need  it  or  not.  EUS  does  not  involve  any  rewriting  of 
software.  A  EUS  module  can  be  written  and  tested  in  far 
less  time  than  it  takes  to  conduct  a  total  rewrite  of  the 
package  or  to  rewrite  the  user  interface. 

The  expression  "EUS”  was  first  used  by  Dr.  Donovan 
Young  of  the  Georgia  Institute  of  Technology  in  1990 
(Young, 1990a) .  While  working  on  the  design  of  user 
interfaces  for  complex  interactive  project  scheduling 
software,  he  discovered  that  a  system  can  be  programmed  to 
monitor  the  context  (location)  and  the  data  state  of  an 
interactive  session.  He  also  proposed  that  such  context  and 
state  information  could  identify  specific  needs  for 
documentation  and  information  that  a  possible  user  might 
have  at  that  point  in  the  program,  and  that  most  of  the 
desired  documentation  is  capable  of  being  automated.  The 
following  discussion  of  EUS  and  its  features  is  largely 
paraphrased,  with  permission,  from  Dr.  Young's  EUS  white 
paper  (Young,  1990a) . 

It  is  first  necessary  to  draw  a  distinction  between 
functional  training  and  software  familiarization.  A  "user- 
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friendly"  module  is  one  for  which  the  user  can  acquire  the 
necessax cm.:iviter  program  familiarization  by  simply 
interacting  with  the  module.  All  computer  programs,  in 
particular  all  Army  installation  support  modules,  can  be 
made  user-friendly.  On  the  other  hand,  the  actual  function 
being  supported  (equipment  issue,  housing  requests)  may 
require  training.  EUS  does  not  teach  the  user  what  to  do 
functionally,  but  provides  a  user-friendly  interface  to  show 
how  the  computer  does  the  function. 

The  purpose  of  EUS  features  is  to  make  an  interface 
sufficiently  user-friendly  so  that  the  novice  or 
intermittent  user  can  easily  answer  all  questions  about 
vocabulary,  interaction  protocols,  software  capabilities, 
available  actions,  data  requirements,  and  standard 
expectations  merely  by  continuing  to  interact  with  the 
program-  A  EUS  module  can  teach  the  user  everything  about 
its  own  behavior.  The  user  must  bring  an  understanding  of 
the  underlying  functional  concepts  of  the  application. 

Embedded  User  Support  consists  of  a  combination  of  one 
or  more  of  the  following  features: 

•  dynamic  help 

•  what-if  data  manipulation 

•  embedded  tutorial 

•  interaction  monitoring 
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1.2. 3.1  Dynamic  Help.  Dynamic  Help  subsystems 
display  state-specific  messages  that  replace  older 

collections  of  error  messages,  status  reports,  and  prompts. 

A  modern  interactive  system  is  almost  always  in  an  input- 
ready  state,  and  at  any  one  time  there  is  a  narrow  range  of 
appropriate  inputs.  A  short  message,  usually  stating  what 
can  now  be  done  by  the  user,  is  on  display  at  all  times. 
Longer,  more  detailed  Help  messages  are  invoked  by  user 
requests.  Both  kinds  of  Help  messages  are  dynamic;  their 
content  depends  on  the  current  state  and  on  data  values. 

The  dynamic  Help  subsystem  builds  messages  as  part  of  each 
interactive  response.  Thus,  the  programmer  or  designer  does 
not  write  specific  messages,  but  only  generic  ones.  The 
computer  will  fill  in  all  the  details  of  the  help  message. 

1.2. 3. 2  What-if  Data  Manipulation.  A  module  that  is 
not  part  of  an  integrated  database  typically  grants  the  user 
too  much  data  destroying  power.  Modification  to  protect 
real  data  and  allow  manipulation  of  what-if  data  can  be 
accomplished  by  changing  the  module's  file  manipulation 
capabilities. 

Modern  practice  is  for  the  system  to  load  a  copy  of  the 
relevant  data  in  local  disk  storage  or  memory,  interact  with 
this,  and  then  perform  a  verification  or  confirmation  step 
to  test  validity  of  the  overall  set  of  transactions  before 
issuing  the  command  to  change  the  real  database.  Protection 
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for  a  training  or  practice  session  is  provided  simply  by 
suppressinq  ...  final  step.  The  what-if  data  feature  is 
useful  not  only  for  training,  but  for  experienced  users  who 
want  to  explore  more  than  one  way  of  accomplishing  a 
transaction.  What-if  data  manipulation  is  essential  in  such 
packages  as  pricing  in  procurement,  routing  in  logistics,  or 
scheduling  in  planning. 

1.2. 3. 3  Embedded  Tutorials.  Embedded  tutorials  differ 
from  ordinary  tutorials  in  that  1)  the  user  can  invoke  the 
tutorial  from  within  the  module,  and  2)  a  serial  path 
through  the  tutorial  is  not  enforced.  While  working  on  a 
job,  if  a  user  comes  to  a  place  where  it  is  desirable  to  use 
an  unfamiliar  feature,  he  can  invoke  the  tutorial,  use  it  to 
learn  the  feature,  and  return  to  the  job.  Unfortunately, 
tutorials  are  very  inefficient  to  program,  requiring  design 
work  and  programming  work  comparable  to  that  of  the  program 
itself. 

1.2. 3. 4  Interaction  Monitoring.  The  final  feature  of 
EUS  systems  is  interaction  monitoring,  which  has  been  called 
"learning  management"  in  the  computer  training  arena.  There 
are  two  alternative  levels  of  interaction  monitoring: 
record/playback,  in  which  a  file  of  user  actions  is  built 
during  a  session  so  that  the  session  can  be  reconstructed; 
and  the  history  file,  in  which  the  sequence  of  major  user 
actions  is  archived  for  later  analysis.  The  former  is  very 
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useful  for  preparing  and  playing  back  demonstration 
sessions,  which  can  be  a  good  training  device.  The  latter 
performs  data  aggregation  and  some  analysis  during  the 
session,  and  is  typically  used  as  an  input  file  for  a 
separate  statistical  analysis  program  that  draws  conclusions 
about  efficacy  and  efficiency  of  the  product  or  of  its  user. 

1.3  The  Installation  Support  Modules  Project 

On  US  Army  installations  (posts,  camps,  and  stations) 
there  are  dozens  of  interactive  software  packages  in  daily 
use.  Many  of  these  packages  are  installation-unique  systems 
that  were  developed  by  the  Major  Army  Command  (MACOM) .  None 
of  these  have  been  developed  and  implemented  in  such  a  way 
as  to  be  truly  capable  of  satisfying  all  horizontal  and 
vertical  integration.  There  is  no  data  sharing  across 
functional  areas.  In  fact,  many  of  these  data  initiatives 
are  redundant  and  are  functionally  duplicative.  The  total 
number  of  interactive  software  packages  used  daily  on  all  US 
Army  Installations  is  estimated  to  be  between  150  to  200 
(Installation  Support  Modules,  [1990]). 

Very  few  installation-unicjue  systems  have  been 
developed  in  accordance  with  a  standard  system  or 
information  architecture.  This  makes  it  difficult  to  deploy 
them  to  other  installations  or  use  them  across  varying 
hardware  platforms.  The  US  Army  Installation  Support 
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Modules  (ISM)  project,  a  high  priority  project,  has  been 
established  ;.o  enhance  installation  management  Army-wide  by 
integrating  tnis  large  family  of  software  into  a  coherent 
system  (Installation  Support  Modules,  [1990]).  ISome  of  the 
goals  of  the  ISM  project  are  to  develop  an  interface  that 
non-ADP  users  will  find  easy  to  use  and  to  develop  a 
standard  data  encyclopedia/dictionary  for  use  at 
installation  level. 

The  ISM  project  is  applicable  at  all  Ammy 
installations.  It  will  support  the  sustaining  software  base 
and  will  interface  with  tactical  and  strategic  systems.  The 
ISM  project  will  provide  installation  commanders  with  an 
Installation  Level  Integrated  Data  Base  (ILIDB)  and  with 
software  to  effectively  manage  daily  operations  and  perform 
the  landlord  functions  outlined  in  AR  5-3,  Installation 
Management  and  Organization.  The  Installation  Software 
Modules  will  reduce  redundant  data  entry  and  duplicative 
input  terminals,  serve  as  input  and  output  mechanisms  for 
the  Standard  Army  Management  Information  Systems  (STAMIS) , 
interface  with  STAMIS  via  the  ILIDB,  and  provide  the 
required  capability  to  share  accurate  and  timely  infoirmation 
at  various  levels  of  installation  management.  Together  ISM 
and  ILIDB  will  help  meet  the  information  needs  of  tenant 
activities  and  higher  command  echelons  and  provide  the 
integration  mechanism  at  installation  level  for  tactical  and 
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strategic  systems  (Installation  Support  Modules,  [1990]). 

1.4  The  Embedded  User  Support  Project 

The  Embedded  User  Support  (EUS)  Project  is  a  project 
within  the  Installation  Support  Modules  Program  which  will 
embed  usability  features  within  two  existing  US  Army 
computer  programs,  minimizing  the  need  to  provide  off-line 
training  and  additional  documentation.  The  project  is  being 
conducted  at  the  Army  Institute  for  Research  in  Management 
Information,  Communications  and  Computer  Sciences  (AIRMICS) 
located  at  The  Georgia  Institute  of  Technology.  The  purpose 
of  incorporating  EUS  capabilities  into  software  systems  is 
to  ease  the  learning  process  for  a  particular  computer  task 
by  not  requiring  a  user  to  recall  a  large  amount  of 
information  at  once.  EUS  standards  will  be  developed  and 
will  incorporate  principles  of  effective  human-computer 
interface  design. 

The  first  phase  of  the  EUS  project  is  to  add  Dynamic 
Help  to  the  interface  for  the  Automated  Central  Issue 
Facility  System  (ACIFS)  program.  The  ACIFS  program  is  DBMS- 
based  (built  around  a  relational  database  management 
system) ,  making  the  context  and  data  state  of  the  system 
very  easy  to  monitor  via  queries  in  the  standard  query 
language  SQL. 
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1.5  The  Research  Problem 


Or<r;»  Help  is  added  to  portions  of  the  ACIFS 

Program,  it  will  be  necessary  to  measure  how  effective  it 
was  in  improving  the  user  interface.  The  purpose  of  this 
thesis  is  to  determine  whether  or  not  the  Dynamic  Help  added 
to  the  ACIFS  Program  was  effective,  and  at  what  places  in 
the  program  it  was  most  effective,  where  it  could  be 
improved  and  where,  if  ever,  it  had  a  negative  effect.  It 
will  do  so  through  the  development  of  measures  of  user- 
support  effectiveness  for  novice  and  intermittent  users,  the 
design  and  conduct  of  an  experiment  to  compare  pre-  and 
post-conversion  effectiveness,  and  the  analysis  of  data 
collected  during  experimentation. 

1.6  Guide  to  this  Thesis 
Chapter  II  reviews  the  background  literature 
surrounding  measures  of  effectiveness  for  user  interfaces 
and  user  testing  of  software. 

Chapter  III  examines  the  measures  of  effectiveness 
which  will  be  collected  during  user  testing  of  the  ACIFS 
Program,  the  identification  of  potential  areas  of  user 
confusion  (challenges) ,  the  creation  of  the  tasks,  the 
conceptual  hypotheses  and  the  possible  sources  of  variation. 

Chapter  IV  describes  the  actual  conduct  of  the 
experiment.  Included  in  it  are  the  data  collection 


13 


procedures,  the  pre-testing  instructions  to  participants  and 
the  administrative  procedure  instructions.  It  also 
describes  the  determination  of  the  number  of  test  subjects 
and  includes  the  specifications  for  automatic  variable 
collection  and  the  resulting  hypotheses  to  be  tested  at  the 
conclusion  of  data  collection  and  describes  the  analysis  of 
the  collected  data  in  the  three  measures-of-ef f ectiveness 
categories . 

Chapter  V  describes  the  results  of  the  tests  of 
hypotheses.  It  then  presents  conclusions  from  the  analysis 
and  generalizes  the  results  of  the  partial  application  of 
Dynamic  Help  to  ACIFS  to  make  conclusions  about  the 
usefulness  of  Dynamic  Help  to  DBMS-based  systems  and 
recommendations  of  diagnostic  improvements.  It  identifies 
prospective  directions  for  further  research  in  Embedded  User 
Support . 
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CHAPTER  II 


LITERATURE  SEARCH 

2.1  User  Interface  Measures  of  Effectiveness 
Gould  (1988)  and  Karat  (1988)  both  emphasize  that 
measurable  behavioral  objectives  are  necessary  to  determine 
whether  or  not  a  particular  computer  program's  user 
interface  is  "good"  or  "bad".  A  literature  search  of 
measures  of  user  interface  effectiveness  yields  measurement 
criteria  options  that  range  from  very  general  to  specific. 
Shneiderman  (1987),  Young,  Miller  and  Coleman  (1987),  and 
Whiteside  (1988)  all  offer  similar  sets  of  user  interface 
effectiveness  measures. 

Ben  Shneiderman  (1987)  defines  five  measures  of  human 
performance:  time  to  learn,  speed  of  performance,  rate  of 

errors  by  users,  user  subjective  satisfaction  and  retention 
over  time.  He  also  explains  that  often  there  are  forced 
tradeoffs  between  the  5  categories  that  may  prevent  success 
in  every  one.  For  instance,  an  emphasis  on  reduction  in 
error  rate  may  affect  the  speed  of  performance.  Designers, 
purchasers,  managers  and  evaluators  must  decide  which  of 
these  measures  are  most  appropriate  for  evaluation  of  their 
particular  system. 
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Young,  Miller  and  Coleman  (1987)  claimed  that  the 
"ultimate  measure  of  the  quality  of  the  DSS  (decision 
support  system)  is  its  performance"  (p.3-43)  and  that 
aspects  of  human  performance  that  are  measurable  in  a  user 
interface  are  accuracy  (error  rate) ,  speed,  training  time, 
and  user  satisfaction.  They  also  state  that  user 
satisfaction  may  be  an  acceptable  substitute  for  the 
accuracy,  speed,  and  training  time  measures  when  resource 
constraints  make  these  measures  difficult  to  obtain. 

Whiteside  (1988)  gives  a  comprehensive  list  of  22 
measurement  criteria  which  includes: 

1.  time  to  complete  a  task 

2.  percent  of  task  completed 

3 .  ratio  of  successes  to  failures 

4 .  time  spent  in  errors 

5 .  percent  or  number  of  errors 

6.  frequency  of  help  and  documentation  use 

7 .  time  spent  using  help  or  documentation 

8.  percent  of  favorable/unfavorable  user  comments 

9 .  number  of  available  commands  not  invoked 

10.  number  of  times  users  need  to  work  around  a 
problem 

11.  number  of  times  the  user  is  disrupted  from  a  work 
task 

12.  number  of  times  user  loses  control  of  the  system. 
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Each  of  these  measures  is  quantifiable  and  objectively 
obtainable,.  7he  specific  measures  to  be  used  in  evaluating 
the  user  interface  quality  must  be  determined  by  the 
managers  and  the  evaluators  of  the  system.  The  measures  are 
normally  determined  by  the  questions  which  are  to  be 
answered  through  testing.  For  example,  if  accuracy  is  of 
primary  importance  and  management  desires  to  know  the 
accuracy  rate  with  a  current  program,  then  frequency  of 
errors  may  be  the  primary  effectiveness  measure.  If  a  new 
help  module  has  been  added  to  a  system,  management  may 
desire  to  know  whether  it  is  being  accessed  by  users.  In 
this  case,  frequency  of  help  and  documentation  use  would 
most  likely  be  the  effectiveness  measure. 

In  Chapter  III,  the  measures  of  effectiveness  derived 
for  this  thesis  will  adhere  to  the  concepts  found  in  the 
above  literature.  The  three  general  categories  to  be 
addressed  are:  speed  (or  productivity) ,  accuracy  and  user 
satisfaction. 

2.2  Methods  of  Measuring  Effectiveness 

There  are  several  ways  to  collect  user  interface  data. 
Data  collection  can  be  perfoinned  for  either  diagnostic  or 
evaluative  purposes.  Diagnostic  testing  identifies  needed 
improvements  to  a  user  interface.  Evaluative  testing 
determines  how  "good"  a  user  interface  is  and/or  identifies 
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the  most  desirable  of  two  or  more  systems  by  means  of  user 
interface  comparison.  While  diagnostic  testing  is  useful  in 
enhancing  a  user  interface,  it  does  not  describe  or  measure 
how  "good"  or  "bad"  the  interface  is.  This  thesis  will 
concentrate  on  evaluative  testing  for  the  primary  purpose  of 
determining  whether  or  not  Dynamic  Help  was  useful. 
Secondarily,  diagnostic  measures  will  be  used  to  determine 
where  and  how  the  Dynamic  Help  implementation  can  be 
improved.  The  literature  on  software  evaluation 
methodologies,  including  Shneiderman  (1987),  Karat  (1988), 
Whiteside  (1988),  and  Young,  Miller  and  Coleman  (1987), 
recommends  a  combination  of  questionnaires  and  controlled 
experiments  for  software  effectiveness  testing. 

Shneiderman  (1987)  states  that  there  are  several  ways 
to  measure  user  performance  and  attitudes.  The  evaluative 
methods  he  describes  are  written  surveys,  interviews, 
controlled  psychologically  oriented  experiments,  and 
continuous  user  performance  data  collection.  He  states  that 
written  surveys  are  "an  inexpensive  and  generally  acceptable 
approach  with  both  management  and  users"  (p.398)  and  may  be 
"useful  in  demonstrating  improvement  to  the  interface  as 
changes  are  made  to  training,  online  assistance,  command 
structures,  and  so  on"  (p.399).  On  interviewing, 

Shneiderman  states  that  "interviews  with  individual  users 
can  be  productive  because  the  interviewer  can  pursue 
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specific  issues  of  concern"  (p.407).  However,  he  also 

states  t'oc  1  "interviewing  can  be  costly  and  time  consuming 

so  usually  only  small  fractions  of  the  user  community  are 

involved"  (p.407).  Controlled  psychologically  oriented 

experimentation  combines  human  factors  with  the  scientific 

method  to  study  an  interactive  system.  Future  management 

decisions  can  be  supported  with  the  quantifiable  results  of 

a  carefully  controlled  experiment.  Shneiderman  suggests 

Fractions  of  the  user  population  could  be  given 
proposed  improvements  for  a  limited  time  and  then 
performance  could  be  compared  with  the  control  group. 
Dependent  measures  could  include  performance  times, 
user-subjective  satisfaction,  error  rates,  and  user 
retention  over  time  (p.412). 

Continuous  user  performance  data  describes  "patterns  of 

system  usage,  speed  of  user  performance,  rate  of  errors,  or 

frequency  of  request  for  online  assistance"  (p.412).  The 

data  can  also  assist  management  in  future  acquisition  and 

training  decisions. 

John  Karat  (1988)  discusses  two  broad  categories  of 
software  evaluation  methodologies:  User-Based  Evaluations 
and  Task-Based  Evaluations.  User-Based  Evaluations  concern 
the  collection  of  data  from  individuals  using  a  software 
system.  Task-Based  Evaluations  are  an  analysis  of  the  task 
which  will  be  performed  with  the  software  and  are  considered 
more  theoretical  in  nature.  In  User-Based  Evaluations, 

Karat  states  that  "surveys,  questionnaires,  and  general 
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descriptive  studies  are  often  the  evaluation  of  choice 
(p.895)"  for  existing  systems.  In  discussing 
questionnaires,  he  states  that  surveys  can  be  of  several 
forms,  but  in  order  to  be  useful  in  evaluation,  he 
recommends  that  survey  questions  be  specific  and  about 
actual  experiences  with  the  software  as  opposed  to  opinions 
on  forecasts  of  possible  software  changes.  He  claims  that 
"the  task  of  answering  the  question  should  not  be  an 
adventure  in  creative  thinking  for  the  subject,  but  should 
rather  request  a  report  of  experience"  (p.896). 

He  mentions  that  evaluations  based  on  "logging  user  behavior 
during  interaction  with  a  software  system"  (pp. 895-896)  are 
more  desirable  in  many  ways  than  questionnaires.  He  states 
that  "Evaluations  by  controlled  experiments  which  are 
designed  to  collect  objective  measurements  such  as  time  to 
complete  a  task,  or  correct  responses  and  errors  are 
commonly  used  in  system  evaluation"  (p.898).  He  also  states 
that  controlled  experiments  focus  more  on  hypothesis  testing 
than  surveys  and  questionnaires,  but  that  questionnaires  may 
also  be  effectively  used  to  collect  data  for  hypothesis 
testing. 

Whiteside  (1988)  suggests  the  following  user  interface 
measurement  operations  which  include  questionnaires  and 
experimental  testing: 

1.  Ask  the  user  to  perfoirm  a  specific  task. 
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2.  Monitor  user  during  free  use  (logging/observing). 

3.  •  ive  user  a  questionnaire. 

4.  interview  user. 

5.  Survey  users. 

6.  Ask  user  for  critical  incidents  revealing 
successes  or  failures. 

Young,  Miller  and  Coleman  (1987)  cite  that  evaluative 
user  testing  may  be  used  to  determine  "how  good  the  DSS 
(decision  support  system)  is,  either  in  terms  of  objective 
performance  measures  or  in  terms  of  user  satisfaction" 
(p.13-9).  The  data  collection  methods  they  suggest  are: 
user  behavior  monitoring,  interviews,  and  questionnaires. 
User  behavior  monitoring  can  be  accomplished  automatically 
through  use  of  the  computer,  by  videotape,  and/or  by 
observational  notetaking. 

Young,  Miller  and  Coleman  (1987)  discuss  two  methods  of 
automatic  data  collection  which  can  be  built  into  the 
computer  system  to  monitor  or  record  user  behavior  or 
performance.  The  first  is  automatic  session  recording  (or 
data  logging)  in  which  each  user  input  act  is  written  to  a 
file  called  a  session  record.  The  second  is  a  history  file 
which  collects  specified  user  input  data  as  well  as  system 
state  values  and  responses  at  predetermined  points  in  the 
software,  with  either  method,  it  is  possible  to  begin/end 
data  collection  with  starting/ stopping  commands  and/or  to 
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make  provisions  for  recording  of  remarks  during  or  after  a 
session.  Interviews  may  be  conducted  either  before  or  after 
a  testing  session  and  questionnaires  may  be  presented  at  the 
determined  appropriate  time. 

Many  system  evaluators  have  used  both  controlled  user 
testing  and  user  questionnaires  as  data  collection  methods 
in  software  evaluation.  Schleske-Peters  (1980)  designed  his 
evaluation  of  an  interactive  generalized  decision  tree 
algorithm  using  methods  of  controlled  user  testing  and 
questionnaires.  Ledgard,  Singer  and  Whiteside  (1981)  also 
chose  the  user  testing/questionnaire  combination  to  evaluate 
user  performance  and  user  satisfaction  between  two  text 
editors.  Villarreal-Ogushi  (1984)  used  the  same  methods 
(user  testing  and  questionnaires)  to  evaluate  efficiency, 
effectiveness,  and  user  satisfaction  in  the  solution  of 
resource-constrained  scheduling  problems  using  GITPASE. 

Also,  during  their  user  tests,  Ledgard,  Singer  and  Whiteside 
as  well  as  Villarreal-Ogushi,  automatically  inserted  the 
task  session  user  inputs  into  a  session  record. 

For  purposes  of  this  experimental  work,  the  author  will 
use  controlled  experimental  testing  and  user  opinion 
questionnaires  to  evaluate  the  usefulness  of  Dynamic  Help. 
Automatic  data  collection  and  note  taking  will  be  used  to 
collect  data  on  user  performance  and  the  questionnaires  will 
be  used  to  collect  data  on  user  satisfaction. 
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2.3  Experimental  Testing  Procedures 
Younq,  .filler  and  Coleman  (1987),  Karat  (1988),  and 
Reisner  (lybj)  all  suggest  steps  necessary  to  conduct 
experimental  testing.  The  following  is  a  synopsis  of  their 
procedures : 

The  basic  procedure  that  Young,  Miller,  and  Coleman 
advocate  in  administering  an  evaluative  user  test  is: 

1.  Select  a  sample  of  users  or  surrogate  users. 

2 .  Have  users  exercise  the  system  to  solve  the 
designed  test  problems. 

3.  Make  observations  to  be  used  in  analysis. 

Karat  (1988)  offers  the  following  steps  as  being 

required  in  experimental  testing: 

1.  Pose  the  question  to  be  answered  by  testing. 

2.  Determine  objective  measures  to  be  taken. 

3.  Design  the  experiment. 

4.  Find  representative  users  to  test. 

5.  Collect  the  data. 

In  discussing  controlled  behavioral  experiments, 
Reisner  (1983)  outlines  an  even  more  comprehensive  set  of 
steps  in  developing  and  conducting  such  experiments: 

1.  Obtain  a  running  system 

2.  Determine  the  purpose  or  hypothesis 

3 .  Determine  the  approach  or  method 

4 .  Design  the  experiment 
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5  . 

Develop 

test  materials 

6. 

Obtain  experimental  subjects 

7  . 

Acquire 

teaching  materials 

8  . 

Acquire 

documentation 

9  . 

Develop 

measurement  tools 

10  . 

"Debug" 

the  experiment 

11 . 

Run  the 

experiment 

12. 

Analyze 

the  data 

13  . 

Perform 

statistical  tests. 

The  author  will  use  a  modified  version  of  Reisner's 
method  to  perform  the  experiment.  One  of  the  goals  of  ISM 
is  that  users  will  be  able  to  use  computer  systems  without 
the  need  for  extensive  training  or  off-line  documentation. 
Therefore,  in  the  development  of  this  experiment,  Reisner's 
steps  7  and  8  will  not  be  used. 
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CHAPTER  III 


METHODOLOGY 

3.1  Objective 

The  primary  aim  of  the  experimental  work  in  this  thesis 
is  to  test  the  effectiveness  of  a  Dynamic  Help  system  that 
was  added  to  the  ACIFS  Program  in  the  first  stage  of  the  EUS 
Project.  The  underlying  evaluative  question  is  whether  or 
not  this  first  application  demonstrates  that  Dynamic  Help 
can  improve  software  usability.  A  secondary  aim  is 
diagnostic;  to  study  how  and  under  what  conditions  Dynamic 
Help  can  influer.ca  user  performance  and  satisfaction. 

To  achieve  these  aims,  it  was  necessary  to  develop  an 
experimental  program  to  identify  those  areas  in  the  ACIFS 
software  where  Dynamic  Help  actually  improved  or  degraded 
usability  and  where  specific  Dynamic  Help  messages  could  be 
improved  for  the  novice  or  intermittent  user.  It  is  assumed 
that  a  user  has  tasks  to  perform,  and  that  in  the 
performance  of  them  certain  challenges  arise  in  which  the 
appropriate  user  actions  are  unclear.  It  is  hypothesized 
that  a  Dynamic  Help  message,  if  effective,  can  provide 
information  that  helps  the  user  overcome  a  challenge.  If 
Dynamic  Help  can  be  shown  to  give  significant  aid  to  users 
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in  overcoming  realistic  challenges  that  commonly  occur  in 
everyday  ACIFS  tasks,  then  the  hypothesized  improvement  in 
software  usability  is  confirmed. 

Effectiveness  of  a  Dynamic  Help  message  can  be  observed 
in  terms  of  the  user's  saving  time,  avoiding  "dead-end" 
points  (described  later),  or  avoiding  errors.  Section  3.4 
will  discuss  the  measures  of  effectiveness  derived  for  this 
experiment . 

This  chapter  also  provides  a  description  of  the  ACIFS 
system  and  the  Dynamic  Help  modification  (Section  3.2), 
creation  of  the  ACIFS  challenge  and  task  specifications 
(Section  3.3),  the  hypotheses  to  be  tested  (Section  3. 5), and 
the  possible  sources  of  variability  (Section  3.6). 

3.2  Description  of  System  and  Modification 
3.2.1  The  Automated  Central  Issue  Facility  Software 

The  Automated  Central  Issue  Facility  System  (ACIFS)  is 
a  computer  program  that  allows  users  to  interactively 
perform  most  of  the  informational  functions  of  a  US  Army 
Central  Issue  Facility  (CIF) .  These  functions  are  listed  in 
Appendix  1.  The  ACIFS  software  package  was  designed, 
produced  and  fielded  by  the  US  Army  Information  Systems 
Command's  Software  Development  Center  -  Atlanta  located  at 
Fort  Gillem,  Georgia.  The  system  is  designed  to  provide 
precise  management  of  organizational  clothing  and  equipment 
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in  a  CIF.  It  is  an  interactive  relational  database  systeiri 
driven  by  uiiiaal  data  entry  from  terminals.  It  provides 
instant  access  to  status  of  property  both  in  the  CIF  and  in 
the  hands  of  soldiers  for  a  given  installation.  At  each 
installation  whose  CIF  uses  ACIFS  (some  installations  use  a 
manual  system,  and  others  use  a  different  computer  program) , 
the  database  includes  only  the  equipment  and  soldiers  served 
by  that  installation's  CIF. 

ACIFS  was  developed  to  run  on  a  platform  consisting  of 
an  Intel  320  Microcomputer  equipped  with  a  144  MG  hard  disk 
drive,  six  Wyse  350  terminals,  four  Mannesmann  Tally  MT86 
serial  printers,  and  one  CITOH  400  LPM  parallel  printer, 
using  the  Xenix  3.5  operating  system,  Informix  SQL  Version 
2.1,  and  Informix  4GL  Development  Version  1.1.  The  ACIFS 
software  interface  has  two  types  of  screens:  menu  selBCtion 
and  data-entry. 

The  menu  screens  were  developed  using  the  Informix  menu 
utility,  and  the  data-entry  screens  were  developed  using  the 
Informix  forms  utility.  Help  is  available  in  the  current 
version  only  at  the  menu  selection  screens  and  is  accessed 
by  pressing  the  FIO  key.  The  Help  is  static  or  "canned"  and 
briefly  describes  the  various  functions  displayed  in  the 
menu. 
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While  in  the  Main  Menu  of  the  ACIFS  program,  one  can 
select  from  a  choice  of  nine  different  submenus: 

GIF  Main  Menu 

1  -  Master  File  Maintenance  Menu 

2  -  Adjustment  Transactions 

3  -  Inventory  Menu 

4  -  As  Required  Reports  Menu 

5  -  End  of  Day  Processing 

6  -  Document  Register  Actions  Menu 

7  -  Clothing  Issues  Process 

8  -  Clothing  Turn  In  Process 

9  -  Temp  Loan/Hand  Receipt  Process 

For  purposes  of  this  evaluation  the  EUS  Project  added  a 
Dynamic  Help  package  to  both  the  menu  screens  and  the  data- 
entry  screens  of  four  of  these  submenu  programs:  2  - 
Adjustment  Transactions,  6  -  Document  Register  Actions  Menu, 
7  -  Clothing  Issues  Process  and  8  -  Clothing  Turn  In 
Process . 

3.2.2  EUS  Dynamic  Help  Specifications 

As  part  of  the  EUS  Project,  Young  specified  the  general 
help  requirements  of  a  user  while  using  the  ACIFS  software 
and  how  Dynamic  Help  messages  might  be  designed  and 
structured  to  meet  those  requirements  (Young,  1990b) .  Based 
upon  these  user  requirements  and  the  proposed  Dynamic  Help 
message  structure,  Young  and  the  thesis  author  designed  two 
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different  structures  for  the  Dynamic  Help  messages:  one  for 
the  menu  s.-j'eens  and  the  other  for  data  entry  screens.  The 
author,  a  supply  officer  in  the  US  Army,  specified  the  logic 
and  content  of  the  Dynamic  Help  messages  for  the  four 
submenu  areas  to  be  addressed  by  the  evaluation. 

In  menu  selection  screens,  it  was  determined  that  there 
were  three  broad  categories  or  reasons  why  a  user  might  need 
help: 

1.  General  Orientation:  help  in  determining  where 
one  came  from,  where  one  is,  and  where  one  can  go 
back  to  (Navigation) . 

2.  Help  in  understanding  the  meaning  of  the  screens 
or  functions  listed  (Meaning  of  Screens  or 
Functions) . 

3.  Help  with  Procedures:  how  to  make  selections,  how 
to  abort  an  entry,  the  effect  of  certain  keys. 

The  menu  Dynamic  Help  message  has  a  global  portion  and  a 
local  portion:  a  global  sentence  in  a  help  message  answers 
a  question  that  could  arise  anywhere  on  the  menu  screen;  a 
local  sentence  refers  only  to  the  highlighted  selection  on 
the  menu.  The  global  portion  provides  a  general  orientation 
of  what  menu  screen  a  user  is  currently  in.  The  local 
portion  describes  what  functions  or  tasks  a  user  can  perform 
if  the  menu  item  currently  highlighted  on  the  screen  is 
selected.  Also,  the  local  portion  of  the  message  lists 


29 


applicable  help  with  procedures,  such  as  how  to  select  the 
highlighted  item.  The  structure  of  the  menu  Dynamic  Help 
messages  consists  of  the  following  sequence: 

Global 

General  Orientation 
Local 

Meaning  of  Screen  or  Function 
Help  with  Procedures 

In  data-entzy  screens,  it  was  determined  that  there 
were  five  categories  or  reasons  why  a  user  might  seek  help: 

1.  Help  in  determining  where  one  is  and  what  one 
could  do  there  (Location  and  Function) 

2 .  Help  in  getting  to  another  screen  or  back  to  a 
previous  screen  (Movement  Between  Screens) 

3 .  Help  in  getting  to  the  next  data  entry  or  back  to 
a  previous  data  entry  (Movement  Within  Screen) 

4 .  Help  with  "how  to  enter” :  how  to  complete  the 
entry  in  a  field,  how  to  correct  an  entry,  how  to 
indicate  you  were  finished  with  a  data  entry,  how 
to  deal  with  a  "quirk”,  how  to  abort  a  data  entry, 
the  effect  of  certain  keys  (How  to  Enter) 

5 .  Help  with  "what  to  enter” :  meaning  of  the  entry 
in  a  field,  range  of  acceptable  entries  in  a 
field,  where  to  find  a  datum  on  a  supporting 
document,  examples  of  possible  entries,  formats  of 
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entries  (What  to  Enter) . 

The  ooca-entry  screen  Dynamic  Help  message  also 
contains  global  and  local  portions.  The  global  portion  of 
the  message  applies  to  the  entire  data-entry  screen,  and  the 
local  portion  of  the  message  refers  only  to  the  field  where 
the  cursor  is  located.  The  global  portion  provides  the  user 
with  information  on  location  and  function  of  the  screen, 
movement  between  screens,  and  how  to  enter  data.  The  local 
portion  provided  the  user  with  information  on  the  location 
or  field  that  the  cursor  is  in,  movement  within  the  screen, 
what  data  to  enter,  and  how  to  enter  data.  The  structure  of 
the  data-entry  Dynamic  Help  messages  consists  of  the 
following  sequence: 

Global 

Location  and  Function 
Movement  between  Screens 
How  to  Enter 
Local 

Location  and  Function 
Movement  within  Screen 
What  to  Enter 
How  to  Enter 

Appendix  2 ,  which  reproduces  Sections  2 ,  3  and  4  of 
"Specifications  of  User  Requirements,  and  Dynamic  Help 
System  Standards  for  EUS  Project  and  GIF  Conversion"  (Young, 
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1990b) ,  explains  the  architecture  and  format  of  Dynamic  Help 
messages  in  more  detail. 

Incorporation  of  Dynamic  Help  into  ACIFS  consisted  of  a 
modification  to  existing  ACIFS  program  modules  and  the 
development  of  new  modules  for  invocation  and  display  of 
help  messages.  Specifications  written  by  the  author  were 
issued.  Some  typical  message  specifications  are  reproduced 
in  Appendix  3.  Both  modification  and  new  development  was 
written  by  Chris  Smith  of  the  Georgia  Tech  Software 
Engineering  Research  Center  in  the  Informix-4GL  database 
programming  language,  version  1.1.  Program  development  was 
performed  on  an  AT&T  3B2  machine  with  a  System  V  Unix 
version  3.2.2  operating  system  located  in  the  Georgia  Tech 
Department  of  Electrical  Engineering's  AT&T  Laboratory. 

3.3  Challenge  and  Task  Specifications 

The  four  submenu  functional  areas  (Adjustment 
Transactions,  Document  Register  Actions,  Clothing  Issues 
Process,  and  Turn  In  Process)  were  chosen  to  be  modified  for 
the  evaluation  because  they  described  the  most  common 
nontrivial  tasks  performed  daily  in  a  CIF. 

Let  a  challenge  be  defined  as  an  instance  in  a  task 
performance  where  the  potential  exists  for  the  user  to  be 
confused  about  the  appropriate  action  to  take.  For  ACIFS, 
most  challenges  are  menu  selection  challenges,  which  occur 
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when  the  user  can  be  confused  about  which  function  to  select 


to  comp.letf'  the  task,  or  data-entry  challenges,  which  occur 
when  the  user  can  be  confused  about  how  to  enter  or  what  to 
enter  as  well  as  how  to  navigate  through  the  screen. 

There  are  three  reasons  motivating  the  definition  and 
specification  of  tasks  to  be  performed  during  the 
evaluation.  First,  a  task  is  a  vehicle  for  presenting 
challenges  to  the  user  in  an  organized  manner.  Secondly,  it 
is  an  event  with  a  well-defined  start  and  finish  which 
facilitates  the  production  of  measurable  and  meaningful  data 
about  the  challenges  from  which  user  performance  and 
satisfaction  can  be  determined.  Lastly,  given  a  task  that 
includes  anticipated  challenges,  the  assertion  that  Dynamic 
Help  improves  software  usability  can  be  tested  by  comparing 
performance  of  the  task  with  and  without  Dynamic  Help  (two 
versions  of  the  same  task,  each  with  different  data) .  The 
performance  of  a  single  task  or  a  subtask  in  ACIFS  follows 
the  following  generic  structure: 

1.  Navigation  to  a  screen. 

2.  Performance  of  a  data  entry  or  sequence  of 
entries. 

3.  Screen  closure  to  be  followed  by  navigation  back 
to  a  menu. 

Therefore,  let  a  task  be  defined  as  a  single  job  or  a 
sequence  of  sub jobs  or  subtasks  (at  most  three)  that  are 
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performed  in  a  single  submenu  (set  of  related  screens)  of 
the  ACIFS  Program.  For  instance,  a  task  may  instruct  the 
user  to  perform  a  clothing  turn-in  (a  single  job)  which  is 
to  be  accomplished  within  the  Clothing  Turn  In  Process 
submenu.  Or  a  task  could  instruct  the  user  to  make  three 
different  adjustments  to  the  property  book  (a  sequence  of 
subtasks)  which  are  all  to  be  performed  within  the 
Adjustment  Transactions  submenu. 

3.3.1  Challenge  Identification 

By  performing  common  ACIFS  functions  within  the  four 
submenu  areas,  the  author  identified  and  listed  the  areas 
with  greatest  potential  for  confusing  a  novice  or 
intermittent  user.  This  prepared  the  author  to  be  able  to 
define  realistic  tasks  whose  performance  would  be  expected 
to  improve  through  the  use  of  Dynamic  Help.  It  was  noted 
that  some  challenges,  such  as  confusion  about  the  proper 
format  of  a  hat  size  entry,  could  occur  in  many  different 
tasks.  It  was  necessary  for  the  challenges  to  be  varied  so 
that  most  of  the  Dynamic  Help  message  categories  (the  3  for 
menu  screens  and  the  5  for  data  entry  screens)  could  be 
tested  for  helpfulness,  once  all  potential  challenges  were 
identified,  then  only  the  most  promising  ones  were  selected 
to  be  used  in  the  creation  of  tasks.  The  following  is  a 
list  of  the  most  promising  challenges  that  were  discovered 
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by  the  author  in  the  four  submenu  areas: 

1.  ."pecial  Issue  Military  Occupational  Specialty 

^muS) :  How  does  a  user  determine  which  MOSs  are 

authorized  a  special  issue? 

2.  Clothing  Size:  The  format  for  certain  clothing 
items  is  unclear. 

3.  F1/F2  Keys:  A  user  may  have  confusion  on  how  to 
use  these  two  keys  to  add  or  delete  lines  in  the 
Manual -Turn-In  Items-Returned  (Man  Tl-Itms  Rtnd) 
Worksheet. 

4.  Social  Security  Number  (SSN) :  The  format  for  the 
SSN  entry  may  be  unclear  to  the  user. 

5.  Unit  Identification  Code  (UIC) :  The  user  could 
possibly  enter  "o"  for  zero  or  vice 

versa,  or  could  confuse  the  UIC  with  a  different 
datum . 

6.  Grade:  The  user  may  have  confusion  as  to  the 
level  of  detail  or  the  format. 

7.  Transferred  Items:  The  user  may  not  clearly 
understand  that  in  order  to  process  a  record  after 
inputting  transferrable  items,  one  must  use  the 
escape  (ESC)  Key. 

8.  Condition  Code:  It  may  not  be  readily  apparent 
which  condition  code  to  enter  for  a  Defense 
Resource  Management  Office  Turn  In  document. 
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9.  ESC  Key:  It  may  not  be  clear  to  the  user  that  the 
ESC  Key  must  be  pressed  to  signal  completion  of 
certain  portions  of  a  worksheet. 

10.  Menu  Selection:  It  may  not  be  clear  which  menu 
item  the  user  should  select  to  accomplish  the 
task. 

11.  The  Total  Price  entry  field  in  the  Cancel-Report- 
of-Survey  Screen:  It  may  be  confusing  as  to  what 
should  be  entered  in  the  Total  Price  field. 

12 .  Required  Fields  in  order  to  create  a  Supply 

Requisition:  It  is  unclear  which  fields  are 

required  to  have  entries  in  order  for  a 
requisition  to  be  processed.  As  a  result,  if  the 
user  has  simple  requisitions  to  create  and  is  in  a 
hurry  it  might  be  useful  to  know  the  shortest  way 
through  a  requisition. 

13.  Corrections  in  a  Worksheet:  Once  a  mistake  is 
made  on  the  screen,  it  is  unclear  how  to  correct 
the  error  (to  abort  or  to  try  to  backtrack) . 

14.  Negative  Receipt;  It  is  unclear  how  to  reverse  a 
document  posting.  A  user  may  not  realize  that  the 
Post  Receipts  Function  may  be  used  to  correct 
errors  made  in  posting  documents. 

15.  Reopen  Closed  Documents:  The  error  message  which 
results  when' trying  to  post  a  closed  document  can 
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leave  the  user  unsure  of  what  to  do  in  order  to  be 


able  to  post  the  document. 

16.  Fastest  Movement  through  the  Complete  the  Issue 

Worksheet:  It  is  unclear  how  to  move  quickly 

through  the  screen  without  trial  and  error. 

17.  Non-Issue  Quantity  postings  in  the  Issue  Due-out 
Items  Screen:  The  user  may  not  realize  that  all 
listed  items  that  are  not  being  issued  at  the  time 
must  have  a  zero  entry  put  in  the  DOQTY  column. 

3.3.2  Task  Specifications 

A  well-conceived  task  is  realistic  (the  scenario  is 
similar  to  actual  tasks  users  typically  perform)  and  common 
(the  task  requirement  appears  often  in  the  real  world) . 

Also,  it  contains  a  high  density  of  promising  challenges. 

The  most  severe  challenges  identified  were  used  to  create 
tasks  (either  as  a  single  job  or  a  sequence  of  subtasks) . 

The  tasks  were  designed  to  be  of  sufficient  length  to  allow 
the  user  to  reasonably  exercise  the  system  in  the  submenu 
area,  yet  not  be  so  long  as  to  cause  user  fatigue  and 
carelessness. 

Several  tasks  were  designed  around  the  identified 
challenges.  They  were  then  screened  for  deficiencies  and 
the  list  of  tasks  was  refined  and  updated  as  follows: 

1.  Candidate  challenges  were  eliminated  when  they 
turned  out  to  be  already  familiar  to  the  user, 
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very  well  documented,  or  obvious,  or  easily 
overcome  by  trial  and  error. 

2.  Inessential  tedious  portions  of  tasks  were 
eliminated  to  increase  the  density  of  the 
challenges  remaining. 

3 .  The  tasks  were  restructured  to  approximately 
balance  performance  times. 

4 .  The  tasks  were  restructured  to  stay  within  a 
single  submenu  to  eliminate  unplanned  navigation 
confusions . 

In  order  to  obtain  an  indication  of  how  challenging  or  time- 
consuming  each  task  would  be  to  potential  test  subjects,  it 
may  be  desirable  to  conduct  a  pilot  test  in  order  to  further 
refine  the  task  descriptions  prior  to  the  actual  experiment. 

The  following  is  a  description  of  the  five  resulting 
base  tasks  (which  in  the  experiment  were  identified  by  the 
numbers  11  through  15)  : 

1.  Task  11,  a  single  task  under  the  Clothing  Turn  In 
Process  Menu,  consists  of  the  posting  of  a  manual 
turn-in  of  a  soldier's  clothing  and  equipment. 

2 .  Task  12 ,  a  sequence  of  three  subtasks  under  the 
Adjustment  Transactions  Menu,  consists  of: 

12.1  Posting  a  turn-in  to  DRMO. 

12.2  Posting  a  statement  of  Charges  (S/C). 

12.3  Modify  a  Report  of  Survey  (R/S) . 
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Task  13,  a  sequence  of  three  subtasks  under  the 
Document  Register  Actions  Menu,  consists  of: 


13.1  Create  four  supply  requisitions. 

13.2  Create  a  supply  requisition  from  an 
unreadable  National  Stock  Number  (NSN) . 

13.3  Post  a  local  cancellation  to  a 
requisition. 

4.  Task  14,  a  sequence  of  two  subtasks  under  the 
Document  Register  Actions  Menu,  consists  of: 

14.1  Posting  a  receipt. 

14.2  Correcting  an  incorrect  receipt  posting 

5.  Task  15,  a  sequence  of  two  subtasks  under  the 
Clothing  Issues  Process  Menu,  consists  of: 

15.1  Post  a  completed  clothing  issue. 

15.2  Post  an  issue  of  due-out  clothing  items 

3.4  Measures  of  Effectiveness 

Ideally,  the  measures  of  effectiveness  to  be  used  in 
the  analysis  should  be  robust  quantitative  expressions  of 
how  usable  the  software  is,  as  indicated  either  by  a 
resultant  improvement  in  user  performance  or  by  user 
opinions  that  indicate  perceptions  of  interface 
friendliness.  Therefore,  measures  of  effectiveness  will  be 
categorized  under  user  performance  and  user  satisfaction. 
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The  measures  defined  below  are  based  on  assuming  that 
when  the  user  occasionally  encounters  a  challenge  in  which 
it  is  unclear  how  to  proceed,  the  available  actions  are: 

1.  To  experiment  by  trial  and  error,  making  entries 
or  selections  intended  to  be  rejected  by  the 
program  or  reversible  by  the  user  if  incorrect, 
and  hoping  that  error  messages  or  recognizably 
inappropriate  system  responses  will  provide 
evidence  as  to  whether  an  entry  or  selection  was 
correct . 

2.  To  invoke  Dynamic  Help,  if  available. 

3.  To  give  up,  leaving  the  task  incomplete. 

4.  To  cheat  or  temporize,  making  an  entry  known  to  be 
incorrect  but  hoped  to  be  accepted  by  the  program. 

As  an  example,  if  a  user  must  enter  the  hat  size  known  as 
"six  and  seven-eighths,"  such  entries  as  "6-7/8"  or  "0678" 
might  constitute  trial -and-error  experimentation;  invoking 
Dynamic  Help,  if  available,  would  yield  a  message  showing 
that  "6875"  is  the  appropriate  entry;  giving  up  would 
constitute  asking  questions  of  other  people  or  consulting 
written  documentation;  and  entering  a  wrong  but  acceptable 
size  such  as  "7"  might  constitute  cheating  (or  temporizing, 
if  the  user  intended  to  correct  the  entry  later) . 
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3.4.1  User  Perfonnance 


Two  types  of  attributes  characterize  user  performance 
in  ACIFS  tasks:  quality  of  performance  and  productivity. 
Quality  of  performance  is  usually  described  by  accuracy  or 
error  rate  and  productivity  is  usually  described  by  ability 
to  perform  (usability)  and  by  production  rate. 

3.4. 1.1  Accuracy.  In  GIF  Operations,  100%  accuracy  is 
desired  in  proper  inventory  and  accounting  procedures  with 
expert  users  of  the  system.  In  those  cases,  one  could 
measure  the  rate  of  transaction  completion  (speed)  without 
errors.  However,  when  novice  or  intermittent  users  operate 
the  system,  a  100%  accuracy  standard  is  unrealistic,  and 
some  measurement  of  user  accuracy  should  be  taken.  Since 
the  relative  seriousness  of  various  kinds  of  errors  is 
difficult  to  define,  and  since  no  baseline  accuracy  measure 
exists  for  a  novice  or  intermittent  user,  a  practical 
measure  of  accuracy  is  simply  the  number  of  errors  a  user 
makes.  Of  interest  is  how  the  addition  of  Dynamic  Help 
affects  the  number  of  errors  a  user  makes  in  task 
performance.  An  error  is  defined  as  any  one  of  the 
following: 

1.  Erroneous  omission  or  addition  of  clothing  items 
to  a  soldier's  clothing  record. 

2.  Incorrect  responses  to  queries  about  soldiers. 

3 .  Incorrect  posting  of  information  into  on-screen 
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documents. 


Total  Task  Errors  will  be  expressed  in  whole  numbers  per 
user  per  task  per  version  (one  version  with  and  one  without 
Dynamic  Help) . 

3.4. 1.2  Ability  to  Perform  (Usability^  It  is 
possible  for  a  place  in  the  program  to  be  so  confusing  that 
the  user  becomes  frustrated  and  gives  up,  either  committing 
an  error  (by  omitting  a  data  entry  or  entering  an  acceptable 
but  incorrect  datum)  or  interrupting  the  task  to  seek 
personal  assistance  or  consult  written  documentation.  Since 
the  Army  would  like  to  reduce  or  eliminate  costly  software 
training,  freedom  from  users'  "giving  up"  is  an  important 
characteristic  to  measure.  No  means  of  separating 
frustration-caused  "give-up"  errors  from  ordinary  "mistake" 
errors  was  found;  however,  the  "dead-ends"  described  below 
are  intended  both  to  prevent  "give-up"  errors  and  to  prevent 
huge  frustration-interruption  times  from  distorting 
production-rate  data. 

3. 4. 1.3  Production  Rate.  Given  that  the  ability  to 
perform  is  present,  management  personnel  are  concerned  with 
user  production  rates.  They  would  not  desire  a  modified 
system  which  decreases  user  productivity.  If  at  all 
possible,  they  desire  a  system  which  facilitates  an  increase 
in  1)  user  productivity  and  2)  user  satisfaction  (in  that 
order) .  They  might  accept  a  system  which  increased  user 
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satisfaction  without  affecting  operation  speed  because  of 
the  perceived  benefits  of  having  a  user-accepted  system. 

For  ACIFS  user  testing,  two  productivity  variables  will  be 
measured :  speed  and  loss  of  control . 

There  are  several  measures  of  speed  in  user  testing. 

One  measure  is  to  calculate  the  percentage  of  task 
completion  in  a  fixed  time.  This  measure  has  been  used  in 
comparisons  of  text  editors  where  a  document  is  to  be  edited 
and  the  percentage  of  document  completed  (and  therefore  task 
completed)  is  a  straightforward  calculation  (Ledgard,  Singer 
and  Whiteside,  1981) .  A  more  common  measure  is  to  record 
total  task  times  from  predefined  starting  points  to 
finishing  points.  This  measure  seems  more  appropriate  in 
evaluating  the  effect  of  Dynamic  Help  in  the  ACIFS  software 
because  of  the  variability  in  task  type  lengths  and  the 
complex  set  of  solution  methods.  Also,  it  would  not  be 
meaningful  to  count  extremely  long  performance  times  caused 
by  a  lack  of  user  support,  because  under  actual  job 
conditions,  users  would  eventually  consult  written 
documentation  or  take  some  other  action  rather  than  spending 
time  in  trial  and  error  or  Dynamic  Help  inspection.  The 
units  of  Total  Task  Time  will  be  calculated  in  seconds  per 
user  per  task  per  version  (one  version  with  and  one  without 
Dynamic  Help) . 
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While  becoming  familiar  with  the  ACIFS  Program,  the 
author  discovered  that  there  were  several  places  in  the 
program  where  a  novice  or  intermittent  user  might  become 
"stuck"  and  would  not  be  able  to  complete  the  task  without  a 
manual  or  supervisor  to  refer  to.  Also,  it  was  noted  that 
there  seemed  to  be  no  pattern  to  these  "dead-end"  or  loss  of 
control  points.  One  could  encounter  several  of  these  in  the 
performance  of  one  task  or  may  not  hit  any  of  them  at  all  in 
several  tasks.  In  every  case,  though,  use  of  a  manual  or 
supervisor  for  reference  seemed  necessary  for  satisfactory 
task  continuance;  alternatively,  it  seemed  possible  that  a 
poorly  motivated  user  might  intentionally  commit  an  error  to 
get  past  the  frustration.  One  of  the  important  assumptions 
made  for  purposes  of  this  evaluation  was  that  outside  help 
or  user  support  would  not  be  available  to  the  user,  who 
should  be  able  to  perform  simple  tasks  without  them. 
Therefore,  it  was  found  necessary  to  prevent  an  unusually 
long  task  performance  or  noncompletion  of  a  task  due  to 
"dead-end"  points,  in  order  to  ensure  that  time  and  error 
measurements  are  realistic  and  that  all  users  experience 
task  completion.  In  order  to  capture  these  "dead-end" 
points  while  at  the  same  time  allowing  the  user  to  continue 
performance  of  a  task,  let  a  Dead  End  be  defined  as  that 
point  in  a  task  where  the  user  is  confused  about  how  to  use 
the  computer  to  do  the  task  and  has  unsuccessfully  tried  the 
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following: 

1.  Used  Dynamic  Help  (if  allowed) 

2.  Trial  and  error  (experimenting) 

3.  searched  the  screen  for  instructions. 

Upon  reaching  this  point,  if  the  user  thinks  that  it  would 
still  take  a  whole  minute  or  more  to  discover  how  to  proceed 
with  the  task,  then  the  user  declares  a  Dead  End.  The  Test 
Administrator  (person  administering  the  experiment)  notes 
the  Dead  End  on  a  task  observation  worksheet  and  then 
provides  the  minimum  assistance  necessary  for  task 
continuance.  Because  the  Dead  End  variable  is  dependent  on 
the  user's  subjective  evaluation  of  difficulty  and  time,  it 
is  unknown  how  it  will  affect  Total  Task  Errors  and  Total 
Task  Time.  It  should  be  noted  that  when  a  Dead  End  is 
declared  by  a  user,  virtually  no  user  support  time  will  be 
added  to  performance  time  as  it  would  be  in  an  actual  job 
setting.  It  is  expected  that  a  task  with  many  Dead  Ends 
would  take  a  lot  longer  to  complete  in  an  actual  job  setting 
because  the  user  would  have  to  ask  a  supervisor  for  help  or 
look  it  up  themselves  in  a  manual  (if  user  support  were 
available) .  It  is  hoped  that  the  declaration  of  a  Dead  End 
variable  in  the  design  will  prevent  frustration-caused 
intentional  errors  and  will  allow  more  conservative  time 
differences  between  tasks  done  with  and  without  Dynamic 
Help.  Also,  the  Dead  End  variable  definition  will  allow  for 
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better  test  planning  and  scheduling,  preventing  unusually 
long  task  times  having  huge  productivity  gaps.  Dead  Ends 
will  be  expressed  in  whole  numbers  per  user  per  task  per 
version  (one  version  with  and  one  without  Dynamic  Help) . 

3.4.2  User  Satisfaction 

User  perceptions  of  a  computer  program  can  be 
categorized  as  diagnostic  or  evaluative.  They  can  be 
collected  both  during  and  after  software  testing  through 
observations  and  questionnaires.  Diagnostic  perceptions  can 
be  collected  during  user  testing  in  the  form  of  observations 
of  user  comments  during  tasks  and  questionnaires  between 
tasks.  They  may  also  be  collected  in  a  diagnostic 
questionnaire  or  interview  once  testing  is  completed.  The 
diagnostic  observations  may  then  be  used  to  identify 
strengths  and  weaknesses  in  the  Dynamic  Help  messages. 
Evaluative  user  perceptions  can  be  collected  as  part  of  a 
post-testing  questionnaire  and  should  be  design  comparative 
in  nature.  For  ACIFS  user  testing,  the  evaluative  user 
satisfaction  variable  will  be  defined  as  the  user's  opinion 
of  how  helpful  the  program  with  Dynamic  Help  was  compared  to 
the  program  without  Dynamic  Help.  It  will  be  measured  on  a 
5-point  scale  (1  =  Very  Helpful,  2  =  Somewhat  Helpful,  3  = 
No  Difference  in  Help  (meaning  no  help  at  all) ,  4  =  Somewhat 
Unhelpful,  and  5  =  Very  Unhelpful  (meaning  it  had  a  very 
detrimental  effect) ) . 
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3.5  Conceptual  Hypotheses 

The  general  questions  to  be  answered  through  this 
evaluation  can  be  categorized  under  user  performance  and 
user  satisfaction. 

3.5.1  User  Performance 

General  user  performance  questions  that  should  be 
answered  through  testing  are: 

1.  Is  there  any  difference  in  Total  Test  Time  (all 
tasks)  between  the  software  versions  with  and 
without  Dynamic  Help? 

2.  Is  there  any  difference  in  Total  Test  Errors  (all 
tasks)  between  the  software  versions  with  and 
without  Dynamic  Help? 

3.  Is  there  any  difference  in  Total  Test  Dead  Ends 
(all  tasks)  between  the  software  versions  with  and 
without  Dynamic  Help? 

4.  Is  there  any  difference  in  Total  Task  Time  between 
the  software  versions  with  and  without  Dynamic 
Help? 

5.  Is  there  any  difference  in  Total  Task  Errors 
between  the  software  versions  with  and  without 
Dynamic  Help? 

6.  Is  there  any  difference  in  Total  Task  Dead-ends 
between  the  software  versions  with  and  without 
Dynamic  Help? 
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7.  Is  there  any  difference  in  Challenge  Time  (time  to 
completion  for  each  specific  challenge)  between 
the  software  versions  with  and  without  Dynamic 
Help? 

8.  Is  there  any  difference  in  Number  of  Challenge 
Errors  (for  each  specific  challenge)  between  the 
software  versions  with  and  without  Dynamic  Help? 

9.  Is  there  any  difference  in  Number  of  Challenge 
Dead  Ends  (for  each  specific  challenge)  between 
the  software  versions  with  and  without  Dynamic 
Help? 

10.  Did  users  use  Dynamic  Help  when  it  was  available? 

11.  Which  Dynamic  Help  messages  (if  any)  had  a 
negative  impact  on  performance? 

12.  Once  Dynamic  Help  is  called  for  a  specific 
challenge,  what  is  the  average  time  it  takes  to 
complete  an  entry? 

3.5.2  User  Satisfaction 

General  user  satisfaction  questions  which  should  be 
answered  through  testing  and  questionnaires  are; 

1.  Is  there  any  difference  in  User  Satisfaction 
between  the  software  versions  with  and  without 
Dynamic  Help? 

2.  What  areas  in  the  program  caused  confusion  when 
Dynamic  Help  was  not  available? 
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3.  When  Dynamic  Help  was  available  and  used,  how 
helpful  was  it  to  the  user? 

3.6  Sources  of  Variability 

The  aim  of  the  evaluation  is  to  focus  on  the 
differences  in  friendliness  in  the  interface  with  and 
without  Dynamic  Help.  Therefore  it  is  necessary  to 
eliminate  any  other  sources  of  variability  that  cannot  be 
controlled. 

0ser  behavior  and  performance  depend  on  many  factors, 
some  of  which  can  be  blocked  out  in  careful  design  of  the 
experiment.  Some  of  these  sources  of  variability  are: 

1.  User  familiarity  with  the  task. 

2.  User  familiarity  with  computers  in  general. 

3.  User  familiarity  with  a  GIF  computer  program  other 
than  ACIFS. 

4.  User  familiarity  with  the  ACIFS  software  package 
based  upon  previous  use. 

5.  User  skill  in  overcoming  interface  deficiencies. 

€.  Non-interface  aspects  of  the  software  design,  ie. 

data  structure,  speed,  etc. 

7.  User  familiarity  with  the  ACIFS  software  based  on 
recent  experience,  ie.  learning  during  testing. 

An  initial  step  in  the  design  process  is  to  define  and 
plan  for  the  target  population  which  will  eventually  use  the 
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software.  In  this  case,  the  target  population  consists  of 
military  and  civilian  clerk  personnel  who  are  currently 
novice  or  intermittent  users  of  the  ACIFS  software.  They 
are  not  computer  experts,  yet  they  may  have  worked  with  a 
computer  in  clerk  tasks  and  are  not  considered  to  be 
computer  novices.  Also,  they  are  familiar  with  the  supply 
task  they  desire  to  perform,  although  they  may  not  remember 
exactly  how  to  use  the  computer  to  accomplish  the  task.  In 
this  case,  they  are  familiar  with  simple  supply  tasks  such 
as  clothing  issues,  clothing  turn-ins,  creating  supply 
requisitions,  and  posting  property  book  adjustment 
documents.  However,  they  are  not  necessarily  familiar  with 
high-level  management  tasks  such  as  stock  record  maintenance 
or  inventory  processing. 

Ideally,  all  test  subjects  fit  into  the  target 
population.  To  ensure  this  occurs,  careful  specification  of 
desired  test  subjects  was  communicated  to  the  Installation 
Support  Modules  Project  Office  via  AIRMICS.  All  selected 
test  subjects  were  to  be  screened  prior  to  testing  to 
determine  their  computer  and  supply  task  skills.  Only  the 
data  from  those  test  subjects  fitting  into  the  prespecified 
population  was  to  be  analyzed.  This  prescreening  process 
should  help  to  filter  out  individuals  who  are  not 
representative  of  the  target  population.  Also,  during  the 
prescreen  phase,  those  potential  test  subjects  who  had  been 


50 


previously  exposed  to  the  ACIFS  software  were  to  be 
identified  and  omitted  from  the  user  testing.  This 
eliminates  any  possible  variation  due  to  user  familiarity 
with  the  ACIFS  software. 

An  additional  source  of  variation  is  user  familiarity 
with  a  GIF  computer  program  other  than  ACIFS.  There  are  at 
least  two  other  known  CIF  computer  programs  being  used  by 
various  installations  in  the  US  Army:  the  WANG  version  and 
the  TRADOC  version.  User  familiarity  with  another  CIF 
software  might  cause  some  CIF  functions  performed  with  ACIFS 
to  be  easier  or  more  difficult  depending  on  whether  the 
other  CIF  software  treats  those  same  CIF  functions  similarly 
or  differently.  Users  familiar  with  another  CIF  software 
were  to  be  identified  during  the  pre-screen  process  and 
results  by  type  CIF  software  were  to  be  analyzed  as 
necessary. 

To  eliminate  possible  learning  effects  due  to  test 
subjects'  familiarity  with  the  software  after  the  first 
versions  of  each  task  are  completed,  each  user's  task 
version  sequence.  Help  and  Non-Help  sequence  and  overall 
task  sequence  were  randomized.  The  purpose  of  this 
randomization  is  to  block  out  any  learning  effect  that  may 
be  encountered  with  the  software  use. 

An  additional  source  of  variation  which  needs  to  be 
avoided  during  the  experiment  is  the  variability  due  to 
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non-interface  changes  made  to  the  software  design  (such  as 
data  structure  changes,  etc.)*  During  the  evaluation,  the 
focus  is  on  changes  made  to  the  interface  (to  enhance 
user-friendliness)  and  measuring  variables  which  will 
indicate  the  effect  of  those  interface  changes.  Therefore, 
any  non-interface  changes  which  were  made  to  the  program 
were  to  be  disabled  if  possible,  eliminating  the  potential 
variation  in  interface  variables  that  may  result  otherwise. 
If  non-interface  changes  were  somehow  not  disabled,  then 
appropriate  data  collection  rules  and  methods  would  need  to 
be  defined  and  adhered  to. 
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CHAPTER  IV 


EXPERIMENTATION  AND  ANALYSIS 

4.1  Design  of  the  Experiment 
As  stated  in  Chapter  III,  the  primary  aim  of  the 
experimental  work  in  this  thesis  is  to  test  the 
effectiveness  of  a  Dynamic  Help  system  that  was  added  to  the 
ACIFS  Program.  The  evaluative  focus  is  on  determining  if 
Dynamic  Help  can  improve  ACIFS 's  usability,  productivity, 
and  accuracy.  The  secondary  diagnostic  goal  is  to  study  how 
and  under  what  conditions  Dynamic  Help  can  influence  user 
performance  and  satisfaction. 

This  chapter  reports  the  design,  implementation,  and 
analysis  of  user  tests  to  measure  user  performance 
(differences  in  errors,  time,  and  Dead  Ends)  and  user 
satisfaction  by  means  of  user  testing  which  will  consist  of 
both  performance  testing  and  opinion  questionnaires. 

4.1.1  Designed  User  Population 

The  aim  of  the  EUS  Project  is  to  make  a  computer 
program's  interface  sufficiently  user-friendly  (with  Dynamic 
Help)  so  that  the  novice  or  intermittent  user  can  perform 
tasks  without  outside  support.  The  target  population  for 
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ISM  software  consists  of  military  and  civilian  clerk 
personnel  who  are  novice  or  intermittent  users  of  at  least 
one  ISM  computer  program.  They  are  neither  computer  experts 
nor  computer  novices.  They  are  wellversed  in  the  functional 
task  they  desire  to  perform,  but  they  may  not  know  how  to 
use  the  computer  to  accomplish  it. 

In  the  US  Army  there  is  a  large  population  of  CIF- 
employed  personnel  at  military  installations  around  the 
world.  To  secure  the  most  valid  group  of  test  subjects 
which  represents  this  population,  it  is  desirable  to 
identify  actual  GIF  personnel  or  general  supply  personnel 
(familiar  with  the  supply  processes)  to  test  the  ACIFS 
modification.  As  discussed  in  Chapter  III,  to  minimize 
variation  due  to  computer  and  supply  skill,  the  ideal  test 
participant  should  be  fairly  knowledgeable  in  supply  matters 
(but  not  necessarily  an  expert)  and  have  a  low  to  moderate 
familiarity  with  computers  in  general.  In  this  case,  they 
should  be  familiar  with  simple  supply  tasks  such  as  clothing 
issues,  clothing  turn-ins,  creating  supply  requisitions,  and 
posting  property  book  adjustment  documents.  However,  they 
should  not  necessarily  be  familiar  with  high-level 
management  tasks  such  as  stock  record  maintenance  or 
inventory  processing.  Also,  they  would  be  novice  users  of 
the  ACIFS  software,  although  they  might  be  familiar  with 
other  CIF  programs. 
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Because  of  uncertainty  in  the  availability  of  CIF 
personnel  general  military  supply  personnel  in  the 

Atlanta  area  (due  to  Operation  Desert  Storm) ,  the  basic  test 
subject  definition  was  purposely  broad  enough  to  encompass  a 
wide  range  of  military  or  civilian  supply  personnel  who  may 
or  may  not  work  in  a  CIF.  Considering  the  possible  length 
of  time  that  each  participant  would  be  available,  the  test 
length  for  each  user  was  set  at  one  man-day.  Also,  the 
author  determined  that  she  would  be  able  to  control  up  to 
two  users  at  any  one  time.  Because  of  the  forecasted 
shortage  of  available  personnel,  the  maximum  number  of  test 
subjects  forecasted  to  be  available  was  set  at  25.  It  was 
hoped  that  at  least  20  personnel  of  the  right  type  could  be 
found  to  undergo  testing.  The  user  testing  was  designed 
around  the  following  test  subject  specification: 

25  military  or  civilian  personnel  who  fit  the 
following  general  description:  grade  E3  to  E6; 
Military  Occupational  Specialty  76Y,  76V,  76C,  or 
76P;  low  or  moderate  computer  terminal  experience 
either  in  schooling  or  on  the  job. 

A  copy  of  the  user  testing  personnel  and  equipment  request 
submitted  4  February  91  to  AIRMICS  is  given  in  Appendix  4. 

AjJLlI _ Designed  User  Testing 

As  discussed  in  Section  3.3.2  of  Chapter  III,  the 
author  defined  five  base  task  types  based  on  the  most  severe 
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challenges  discovered  in  the  four  submenu  areas.  In  order 
to  directly  measure  the  usefulness  of  Dynamic  Help,  it  was 
determined  that  the  tasks  were  to  be  performed  with  and 
without  Dynamic  Help.  This  would  provide  a  difference  in 
performance  by  user. 

4. 1.2.1  Test  Structure.  Because  of  the  differences  in 
length  and  difficulty  between  task  types,  direct  comparisons 
of  performance  between  tasks  would  be  impractical . 

Therefore,  two  versions  (of  approximately  equal  difficulty) 
of  each  task  were  to  be  created  so  that  direct  performance 
comparisons  by  task  type  could  be  measured.  Test  subjects 
were  to  perform  a  total  of  ten  tasks  (five  with  and  five 
without  Dynamic  Help)  in  one  day.  It  was  planned  that  each 
user  would  perform  five  tasks  in  the  morning  (one  version  of 
each  task  type)  and  five  tasks  in  the  afternoon  (one  version 
of  each  task  type) . 

In  order  to  block  out  learning  effects  which  would 
occur  between  the  morning  and  the  afternoon  performance  of 
the  tasks  and  effects  due  to  task  version  differences,  the 
order  of  performance  of  particular  task  versions  and  order 
of  Dynamic  Help  use  between  users  would  be  randomized. 

Also,  to  nullify  any  user  anticipation  of  task  type  order  in 
the  afternoon  (performing  same  task  type  sequence  in  the 
morning  and  afternoon) ,  the  task  type  sequences  for  morning 
and  afternoon  would  also  be  randomized.  The  resulting  data 
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should  indicate  the  effect  that  Dynamic  Help  had  on 
performance  without  the  effects  of  learning  and  user 
anticipation. 

User  testing  was  designed  to  be  the  vehicle  by  which 
the  user  performance  variables  (Errors,  Speed,  and  Dead 
Ends)  would  be  measured.  Because  each  user  has  a  different 
task  performance  ability  (some  take  longer,  read  more 
slowly,  while  others  are  quicker  on  the  keyboard,  read 
faster,  etc.),  the  differences  between  users  in  the 
performance  of  a  single  task  version  with  or  without  Help 
could  be  so  great  as  to  mask  any  true  difference  due  to  the 
inclusion  of  Dynamic  Help.  Therefore,  the  differences  in 
performance  should  be  measured  within  subjects  and  the 
paired  t-test  method  should  be  used  to  determine  whether 
significance  is  present  in  the  data.  Hines  and  Montgomery 
(1980)  state  that  the  paired  t-test  may  be  used  when  paired 
observations  are  made  and  the  differences  between 
observation  pairs  are  non-horoogeneous .  The  procedure  is  to 
collect  the  data  in  pairs,  and  then  to  analyze  the 
differences  between  the  pairs.  This  will  eliminate  the 
difference  in  performance  between  users  not  caused  by  the 
treatments  (Non-Help  or  Help) . 

4. 1.2. 2  Paired  T-Test.  The  following  is  a  synopsis  of 
the  paired  t-test  method  as  described  by  Hines  and 
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Montgomery  (1980) : 

Let  ,  •X’21)  ,  (^i2f  ^22)  f  ‘  '  f  (^inf  ^2n)  ®  Set  of  paired 

observations  where  it  is  assumed  that  Xj  is  normally 
distributed  with  parameters  fj.^  and  and  Xj  is  normally 
distributed  with  parameters  /Xj  and  02^-  Let  “  ^2jr  j 

=  l,2,...,n  and  assume  that  the  differences  are  normally  and 
independently  distributed  random  variables  with  mean  and 
variance  Testing  the  hypothesis  Hq:  Mi  =  M2  versus  HiZ 

Ml  *  M2  is  equivalent  to  testing 


Hq:  Md  =  0 
Hi:  Md  ’*  0 

The  corresponding  test  statistic  would  be 


3 

Sp/yfn 


where 


75  - 
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and 
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2 

/n 

j-i 

n-1 


are  the  sample  mean  and  variances  of  the  differences. 

Hj,:  /ip  =  0  is  rejected  either  if  t^  >  t„/2.n-i  or  if  t^  < 

-t„/2,n-i*  One-sided  alternatives  are  treated  analogously. 

4. 1.2. 3  Statistical  Hypotheses.  In  general,  errors 
would  be  defined  as  those  incorrect  responses  to  queries, 
incorrect  data  entries  that  remain  uncorrected  at  the  time 
of  data  processing,  and  incorrect  omission  or  inclusion  of 
items  on  soldier  clothing  records  which  a  user  commits 
during  a  period  between  a  predefined  starting  and  finishing 
point.  To  determine  how  Dynamic  Help  performed,  a  total  of 
the  differences  of  the  number  of  errors  by  user  would  be 
measured.  This  would  be  done  by  aggregating  the  differences 
(Non-Help  errors  minus  Help  errors)  of  the  five  different 
task  types  by  user.  Once  Total  Test  Errors  were  calculated 
by  user,  a  paired  t-test  would  be  conducted.  The  Null 
Hypothesis  would  be  that  the  Mean  Difference  in  Errors 
between  the  two  software  versions  is  equal  to  zero.  The 
Alternative  Hypothesis  would  be  that  the  Mean  Difference  in 


59 


Errors  is  greater  than  zero  (ie.  more  errors  without  help). 
The  a-level  at  which  the  one-sided  t-statistic  (t^,)  becomes 
significant  would  then  be  determined  and  reported  (the  P 
value)  . 

The  speed  of  performance  of  a  task  would  be  defined  as 
the  amount  of  time  in  seconds  elapsing  between  a  predefined 
starting  and  stopping  point.  The  differences  in  Total  Test 
Time  would  then  be  calculated  similarly  to  Total  Test  Errors 
as  an  aggregation  of  the  five  task  performance  time 
differences  (Non-Help  minus  Help)  by  user.  Once  the 
differences  in  tasks  had  been  aggregated  by  user,  a  paired 
t-test  would  be  conducted.  The  Null  Hypothesis  would  be 
that  the  Mean  Difference  in  Time  between  the  two  software 
versions  is  equal  to  zero.  The  Alternative  Hypothesis  would 
be  that  the  Mean  Difference  in  Time  is  greater  than  zero 
(ie.  longer  time  without  help) .  The  level  at  which  the  one¬ 
sided  t-statistic  (to)  becomes  significant  would  then  be 
determined  and  reported. 

Let  Dead  Ends  be  as  defined  in  Section  3. 4. 1.3.  Then 
the  number  of  Dead  Ends  occurring  between  a  predefined 
starting  and  finishing  point  determine  the  total  number  of 
Dead  Ends  for  a  particular  task.  The  differences  in  Total 
Test  Dead  Ends  would  then  be  calculated  similarly  to  Total 
Test  Errors  and  Total  Test  Time  as  an  aggregation  of  the 
five  task  Dead  End  differences  (Non-Help  minus  Help)  by 
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version  of  the  task  questionnaire  would  be  completed  after 
the  performance  of  a  task  type  with  Dynamic  Help  available. 
It  would  ask  the  user  whether  or  not  he  had  used  Dynamic 
Help  at  specific  challenge  points,  the  reasons  why  he  had 
used  Dynamic  Help  at  those  points,  and  how  useful  Dynamic 
Help  was  in  helping  him  overcome  the  challenge.  The 
information  obtained  from  Help  questionnaires  could  be  used 
diagnostically  to  identify  specific  areas  of  Dynamic  Help 
success  or  failure.  The  Non-Help  version  of  the  task  type 
questionnaire  would  be  completed  after  performance  of  a  task 
without  Dynamic  Help  available.  It  would  ask  the  user 
whether  or  not  he  had  been  confused  at  the  predetermined 
challenge  points  and  what  had  been  confusing  to  him  about 
the  challenge.  The  responses  obtained  from  the  Non-Help 
questionnaires  could  diagnostically  verify  (or  nullify)  the 
content  of  the  Dynamic  Help  messages  created  for  those 
specific  challenges. 

Once  each  user  had  completed  the  entire  set  of  tasks, 
it  would  be  useful  to  ask  the  users  for  their  overall 
opinions  of  Dynamic  Help.  This  would  be  accomplished  in  a 
post-test  questionnaire  via  a  question  which  would  ask  the 
user  to  compare  ACIFS  with  and  without  Dynamic  Help.  User 
Satisfaction  would  be  calculated  with  the  user  responses  to 
this  post-test  evaluative  question  and  would  be  reported  by 

means  of  the  mean  and  standard  deviation  of  the  responses. 

*  ‘ 
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Also  in  the  post-test  questionnaire,  general  preference 
questions  about  the  Dynamic  Help  message  structure  would  be 
asked  in  order  to  collect  additional  diagnostic  information 
for  the  current  Dynamic  Help  implementation. 

4.2  Implementation  of  the  Experiment 

The  experiment  was  administered  from  20  March  to  5 
April  1991  at  the  Georgia  Institute  of  Technology  campus 
using  military  and  civilian  GIF  personnel  from  various 
installations  in  the  US. 

4.2.1  User  Population 

The  Installation  Support  Modules  Project  funded  the 
travel  for  test  personnel.  A  total  of  26  personnel 
participated  in  testing:  two  personnel  (USERIDs  1  and  2) 
performed  tasks  as  part  of  a  pilot  test  on  20  March  91,  and 
24  personnel  participated  in  the  user  testing  from  21  Mar  to 
5  Apr  91.  Table  4.1  (see  next  page)  describes  the  dates  of 
testing,  the  User  Identification  (USERIDs)  of  test 
personnel,  the  installations  providing  the  test  personnel, 
and  the  versions  of  GIF  software  the  test  personnel  had 
previously  worked  with. 

The  data  for  Users  19  and  21  was  omitted  from  analysis. 
User  19 's  data  from  the  first  set  of  base  tasks  was  not 
recorded  and  she  was  required  to  perform  the  first  set  of 
tasks  again  in  order  that  data  might  be  collected.  Then  she 
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user.  Once  the  differences  in  tasks  had  been  aggregated  by 
user,  a  paired  t-test  would  be  conducted.  The  Null 
Hypothesis  would  be  that  the  Mean  Difference  in  Dead  Ends 
between  the  two  software  versions  is  equal  to  zero.  The 
Alternative  Hypothesis  would  be  that  the  Mean  Difference  in 
Dead  Ends  is  greater  than  zero  (ie.  more  Dead  Ends  without 
help) .  The  level  at  which  the  one-sided  t-statistic  (t^) 
becomes  significant  would  then  be  determined  and  reported. 

4. 1.2. 4  Performance  Data  Collection  Design.  During 
task  performance,  data  from  which  Errors  and  Speed  would  be 
determined  would  be  recorded  automatically  in  a  task  session 
history  file.  Task  history  file  specifications  would  be 
prepared  and  submitted  to  the  programmer  (the  submitted  task 
history  file  specifications  are  enclosed  in  Appendix  8) . 
Also,  two  of  the  five  tasks  would  generate  printed  output 
from  which  partial  Error  measurements  could  be  made.  Also 
during  testing.  Dead  Ends  would  be  recorded  (User/Task/ 
Screen/Field)  by  the  Test  Administrator  (the  author)  at  the 
time  of  their  occurrence  on  an  observation  worksheet  (See 
Appendix  9) . 

It  was  hoped  that  the  collected  data  would  be  robust 
enough  to  be  used  to  generate  secondary  hypotheses  which 
would  explain  interesting  aspects  about  how  Dynamic  Help 
interacts  with  the  user's  performance.  For  instance,  if 
Dynamic  Help  degraded  user  performance  time,  then  it  would 
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be  of  interest  to  know  exactly  in  what  tasks,  subtasks,  and 
challenges  it  slowed  the  user  down.  This  would  allow  for 
speculation  as  to  why  time  was  spent  in  these  identified 
areas  and  would  allow  for  identification  of  possible  means 
of  improving  Dynamic  Help  or  specific  Dynamic  Help  messages. 
4.1.3  Design  of  User  Questionnaires 

Questionnaires  to  be  completed  by  the  users  would  be 
developed  for  three  purposes: 

1.  To  collect  personal  data  (pre-screen) 

2.  To  collect  diagnostic  perceptions  from  the  users 

3 .  To  collect  evaluative  perceptions  from  the  users 
(User  Satisfaction) . 

Users  would  complete  a  pre-screen  questionnaire  in  order  to 
provide  information  on  user  supply  knowledge,  GIF  knowledge, 
GIF  software  experience,  and  general  computer  skill.  Then 
during  testing,  users  would  complete  a  questionnaire  after 
the  performance  of  each  task.  Each  post-task  questionnaire 
would  ask  specific  questions  abcut  the  challenges  that 
should  have  been  encountered  in  the  task  and  whether  or  not 
the  potential  challenges  were  confusing.  Then,  if  Dynamic 
Help  was  to  be  available  during  the  task,  it  would  be  useful 
to  know  whether  or  not  it  was  used  and  helpful.  For  each 
task  type,  there  would  be  a  questionnaire  for  the 
performance  of  the  task  with  Help  and  another  questionnaire 
for  the  performance  of  the  task  without  Help.  The  Help 
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JLE  4.1  Profile  of  Users 


DATE 

USERIDS 

INSTALLATION 

SOFTWARE 

21  MAR  91 

7  &  8 

Ft  Bragg,  NC 

WANG 

22  MAR  91 

■B 

Ft  Bragg,  NC 

WANG 

25  MAR  91 

5  &  6 

Ft  Sam  Houston,  TX 

WANG 

26  MAR  91 

9  &  10 

Ft  Campbell,  KY 

WANG 

27  MAR  91 

11  &  12 

Ft  Carson ,  CO 

WANG 

28  MAR  91 

13  &  14 

Ft  Carson ,  CO 

WANG 

29  MAR  91 

15  &  16 

Ft  Polk,  LA 

Unknown 

1  APR  91 

17  &  18 

Ft  Sam  Houston,  TX 

WANG 

2  APR  91 

19  &  20 

Ft  Campbell,  KY 

WANG 

1  3  APR  91 

21  &  22 

Ft  Stewart ,  GA 

WANG 

4  APR  91 

23  &  24 

Ft  McClellan,  AL 

None 

5  APR  91 

25  &  26 

Ft  Sam  Houston,  TX 

WANG 
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performed  the  second  set  of  tasks  as  usual  during  tha 
afternoon  session.  Since  she  performed  the  base  tasks  a 
total  of  three  times  each,  her  results  on  both  sets  of 
collected  data  (the  second  and  third  performances)  were  not 
included  in  data  analysis. 

User  21  was  unable  to  complete  the  tasks.  A  review  of 
the  Pre-Test  Questionnaire  showed  that  his  level  of  computer 
familiarity  did  not  fall  within  the  range  of  test  personnel 
desired,  and  hence  his  data  is  not  included  in  the  analysis. 

Also,  during  the  performance  of  Task  15AN,  User  20 
apparently  committed  31  posting  errors  for  unknown  reasons 
not  believed  to  be  related  to  predefined  challenges.  The 
maximum  number  of  non-challenge  errors  made  by  any  other 
user  was  1.  This  anomolous  error  data  was  discarded. 

4.2.2  User  Testing 

The  pilot  test  and  main  user  tests  were  to  be  conducted 
at  the  Software  Development  Center  -  Atlanta  at  Fort  Gillem, 
GA  from  20  Mar  to  5  Apr  91.  Due  to  unforeseen  software  and 
hardware  problems,  the  test  site  was  moved  to  the  Georgia 
Tech  campus.  The  data  entry  terminals  used  for  the  main 
user  tests  were  DEC  vtlOOs,  as  opposed  to  the  WYSE  350 
terminals  for  which  the  ACIFS  software  was  developed.  Since 
the  terminals  were  different,  some  minor  modifications  had 
to  be  made  to  the  program,  such  as  function  key  definition 
changes.  However,  these  modifications  had  no  material 
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effect  on  the  user  interface. 


4_t2_.2_il,  Pilot  Test.  The  pilot  test  was  conducted  at 
the  Georgia  Tech  Department  of  Electrical  Engineering's  AT&T 
Laborato'-y  on  20  Mar  91.  As  a  result  of  the  pilot  test,  the 
author  determined  that  Task  14  was  not  challenging  enough  to 
adequately  test  Dynamic  Help.  Also,  it  appeared  necessary 
to  reduce  the  number  of  tasks  to  ensure  complete  performance 
of  all  tasks  by  users  in  one  day.  As  a  result,  the  five 
base  tasks  were  reduced  to  four  (Tasks  li,  12,  13  and  15). 

4. 2. 2. 2  Main  Performance  Test.  The  user  testing  was 
conducted  at  AIRMICS  from  21  Mar  to  5  Apr  91.  During  user 
testing  each  user  was  to  perform  a  total  of  eight  tasks: 
two  versions  of  each  remaining  task  type,  half  with  Help  and 
half  without  Help.  Appendix  5  contains  the  final  versions 
of  the  eight  tasks.  The  daily  testing  schedule  was  as 
follows: 


EVENT 

Initial  Briefing  to  Participants 

Pre-Screen  Questionnaire 

Four  Tasks  w/  Post-Task  Questionnaires 

Lunch 

Four  Tasks  w/  Post-Task  Questionnaires 
Post-Test  Questionnaire 
Release  of  Participants 


TIME  ( approx . ) 
0800-0830 
0830-0900 
0900-1130 
1130-1300 
1300-1530 
1530-1600 
1600-1630 
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Before  testing  began  each  morning,  test  participants 
were  briefed  on  the  purpose  of  the  testing,  the  planned 
sequence  of  the  test  and  other  pertinent  information.  A 
copy  of  the  briefing  information  is  enclosed  in  Appendix  6. 

To  ensure  consistency  of  actions  and  non- interruption 
of  testing  in  case  of  non-availability  of  the  test 
administrator,  a  control  handbook  of  test  administration 
procedures  was  prepared.  A  copy  of  the  control  handbook 
(GIF  Dynamic  Help  Experiment  Handbook)  minus  enclosures  is 
in  Appendix  7 . 

4 . 2 . 2 . 3  Task  Sequence  As  r  ^nv.^it.  There  were  two 
versions  of  each  of  the  four  task  types,  and  either  version 
could  be  performed  with  or  without  Dynamic  Help.  To  ensure 
that  a  sufficient  amount  of  time  elapsed  between  the 
performance  of  like  tasks,  one  version  of  each  task  type  was 
completed  prior  to  the  user's  lunch  break.  The  remaining 
version  of  each  task  type  was  performed  after  lunch.  The 
order  of  task  type  performance  before  lunch  as  well  as  after 
lunch  was  random.  For  each  day  of  testing  there  was  an  odd- 
numbered  user  and  an  even-numbered  user.  For  the  odd- 
numbered  user,  the  version  of  each  task  type  performed 
before  lunch  was  random.  The  even-numbered  user  performed 
the  other  versions  of  each  task  type  before  lunch.  This 
ensured  database  integrity  while  both  users  were  testing. 
During  the  lunch  break  the  database  was  reinitialized. 
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After  lunch,  the  users  performed  the  opposite  versions  of 
each  task  tvne  (the  versions  they  had  not  yet  performed) . 

For  both  users,  the  assignment  of  Help  and  Non-Help  within 
the  task  sequence  before  lunch  was  random,  while  after-lunch 
versions  of  each  task  type  received  the  opposite  assignment 
of  Help  and  Non-Help.  The  random  numbers  used  for  the 
assignment  of  task  types,  versions  and  Help/Non-Help  were 
taken  from  Table  XV  of  Hines  and  Montgomery  (1980) .  The 
actual  sequence  assignments  are  given  in  Appendix  7 . 

4. 2. 2. 4  Data  Collection.  Originally,  Total  Test  Time 
and  Total  Test  Errors  were  to  be  collected  automatically 
during  testing  by  means  of  task  history  database  tables  and 
other  task  data  files  which  were  to  record  specified  user 
inputs  during  testing.  From  the  Task  History  Table  and 
Other  Task  Data  File  for  each  task  and  user,  a  Session 
Closure  Report  was  to  be  generated.  Automatic  data 
collection  as  envisioned  in  the  Task  History  Tables  enclosed 
in  Appendix  8  was  not  implementable  in  the  Informix 
software.  Instead,  a  different  format  was  implemented  from 
which  an  informative  set  of  task  session  records  was 
created.  The  fields  collected  in  the  Task  History  Tables 
were: 

USERID 

INDEX  (Programmers  reference) 

DATE 
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TIME 


SPECIAL  KEY/HELP  DESIGNATION 
MENU  LOCATION 
FIELD  IDENTITY 

OTHER  IMPORTANT  INFORMATION  (Start  Time,  LIN  Numbers,  etc.) 

ACTUAL  FIELD  ENTRY  BY  USER 

Automatic  data  collection  for  a  particular  task  began 
at  the  time  when  the  Header  Call  was  completed  by  the  Test 
Administrator.  The  Header  Call  was  a  subprogram  built  into 
the  program  which  started  the  clock  and  table  entries  for  a 
user  and  task.  Then  during  the  task  performance,  as  each 
field  was  completed,  information  was  written  to  a  temporary 
table.  Upon  completion  of  a  menu  selection  or  upon 
processing  an  entire  screen  of  data,  the  temporary  table  was 
written  to  the  designated  task  history  table.  Upon 
completion  of  a  task,  the  user  informed  the  Test 
Administrator  who  then  made  the  Closure  Call  for  the  task 
session.  The  Closure  Call  was  a  subprogram  which  closed  out 
the  task  session  in  the  task  history  table.  The  Closure 
Call  also  listed  the  start  and  finish  times  for  the  task  on 
the  screen,  allowing  the  Test  Administrator  to  make  note  of 
the  times  on  the  observation  worksheets. 

There  were  programming  difficulties  in  writing  the 
temporary  table  to  the  task  history  table.  Many  of  the 
temporary  tables  were  written  to  the  wrong  task  history 
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teible,  creating  difficulties  in  data  extraction  and 
necessitating  the  recording  of  task  start  and  finish  times 
on  the  obsej-vation  worksheets.  Also,  entire  tasks  for  some 
users  (except  for  start  and  finish  times)  were  not  recorded 
in  the  task  history  tables  for  unknown  reasons  and  therefore 
are  considered  lost  data. 

As  implemented,  the  data  collection  program  was  not 
able  to  capture  the  use  of  Dynamic  Help  at  menu  selection 
screens,  so  testing  about  how  useful  Dynamic  Help  was  for 
menu  selection  was  weakened.  Also,  once  Dynamic  Help  was 
implemented  in  the  menu  screens,  the  static  or  "canned"  Help 
package  was  disabled.  Therefore,  any  conclusions  about  the 
meefulness  of  Dynamic  Help  in  menu  selection  screens  will 
not  be  based  on  comparisons  of  Dynamic  Help  with  "canned" 
lielp,  but  only  on  comparison  against  having  no  Help. 

During  testing.  Dead  Ends  data  was  recorded  by  the  Test 
Administrator  on  the  observation  worksheets.  Users  were 
toriefed  and  provided  with  written  instructions  on  the 
definition  of  a  Dead  End  and  how  to  declare  one  (see 
Appendix  6  for  Instructions  to  Test  Participants) .  At  the 
time  a  user  declared  a  Dead  End,  the  Test  Administrator 
noted  the  Screen/Field  location  on  the  Observation  Worksheet 
for  that  task  and  user.  Then  the  Test  Administrator 
IKTOvided  the  minimum  assistance  necessary  for  the  user  to 
proceed  with  the  task. 
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4.2.3  User  Questionnaires 

User  C[uestionnaires  were  developed,  created  and 
administered  as  discussed  in  Section  4.1.3.  A  copy  of  the 
Pre-Screen  Questionnaire  is  enclosed  in  Appendix  10.  Copies 
of  the  Post-Task  Questionnaires,  designated  as  IIH,  llN, 

12H,  12N,  13H,  13N,  15H  and  15N  (H  =  Help  version  and  N  = 
Non-Help  version) ,  are  in  Appendix  11.  A  copy  of  the  Post- 
Test  Questionnaire  is  enclosed  in  Appendix  12 . 

4.3  Analysis  of  the  Data 

This  section  covers  how  data  analysis  was  performed. 

The  results  are  given  in  the  following  chapter. 

4.3.1  User  Performance 

Many  users  did  not  perform  Task  11  as  originally 
envisioned.  Through  fault  of  the  task  instructions  rather 
than  the  interface,  many  users  failed  to  post  transferred 
items  to  the  Manual -Turn- In  Items-Transf erred  Screen.  As  a 
result,  it  was  decided  to  exclude  Error,  Time,  and  Dead  End 
data  for  that  screen,  and  all  Task  11  measurements  are  taken 
from  the  start  of  Task  11  to  the  processing  point  (ESC  Key) 
of  the  Manual -Turn-In  Items-Returned  Screen.  Data 
extraction  for  each  of  the  performance  variables  is 
discussed  in  the  following  three  subsections. 

4. 3. 1.1  Errors.  Let  errors  be  as  defined  in 
Section  4. 1.2. 2.  Total  Test  Errors  for  each  user  was 
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calculated  and  the  paired  t-test  performed  as  discussed  ih 
Section  4. 1.2. 2.  Error  data  which  has  been  classified  by 
task,  subtask  and  challenge  was  used  for  diagnostic 
analysis. 

For  Task  11,  errors  were  classified  as  ACIFS-generated 
"printout-challenge"  errors,  "printout-other"  errors,  and 
"other-data"  errors  (non-printout  errors  collected  by  the 
Task  History  Table) .  This  was  done  for  diagnostic  reasons. 
Total  errors  for  each  version  of  Task  11  was  determined  by 
summing  these  three  error  types  for  each  user.  Then  the 
difference  in  errors  was  determined  by  subtracting  the  Help 
error  sum  from  the  Non-Help  error  sum,  by  user. 

Task  12  Errors  were  identified  exclusively  by 
inspection  of  user  actions  recorded  in  the  task  history 
tables.  Errors  were  extracted  by  user  for  each  of  the  three 
subtasks  and  sximmed  to  determine  the  task  total  per  version. 
Then  the  Task  12  difference  in  errors  was  calculated  by 
subtracting  the  Help  data  from  the  Non-Help  data  by  user. 

The  Task  13  error  extraction  process  mirrors  that  of 
Task  12  in  that  the  data  were  identified  in  the  task  history 
tables  for  each  of  the  three  subtasks  and  calculated  in  a 
fashior  similar  to  Task  12. 

Task  15,  which  consisted  of  two  subtasks,  had  its 
errors  determined  by  ACIFS-generated  printouts.  Subtask  I's 
errors  were  classified  as  "challenge"  errors  and  "other" 
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errors.  Subtask  2's  errors  consisted  only  of  challenge 
errors.  Total  errors  for  each  version  of  Task  15  were 
determined  by  totalling  the  Subtask  1  and  Subtask  2  errors 
for  each  user.  Then  the  difference  in  errors  for  each  user 
was  determined  by  subtracting  the  Help  data  from  the  Non- 
Help  data. 

4.3. 1.2  Dead  Ends.  Let  Dead  Ends  be  as  defined  in 
Section  3. 4. 1.3.  Total  Test  Dead  Ends  for  each  user  was 
calculated  and  the  paired  t-test  performed  as  discussed  in 
Section  4. 1.2.2.  Because  Dead  Ends  were  recorded  by  screen 
and  field  location,  their  user  totals  by  task  as  well  as  by 
challenge  and  subtask  (for  diagnostic  purposes)  were  easily 
extracted  from  the  observation  worksheets. 

4 . 3 . 1 . 3  Time .  Let  the  time  of  an  event  remain  as 
defined  in  Section  4. 1.2. 2.  Unless  otherwise  specified,  the 
starting  and  stopping  point  times  for  a  task  are  identified 
by  the  recorded  Header  and  Closure  Call  times.  Total  Test 
Time  for  each  user  was  calculated  and  the  paired  t-test 
performed  as  discussed  in  Section  4 . 1 . 2 . 2 . 

The  time  definitions  of  other  diagnostic  events  are  as 
follows.  For  subtasks,  let  the  starting  point  be  defined  as 
either  the  beginning  of  a  task  (for  subtask  1)  or  as  the 
stopping  point  of  the  previous  subtask.  Let  a  subtask 
stopping  point  be  normally  defined  as  the  time  when  the 
final  data  screen  for  the  subtask  is  processed.  The  usual 
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stopping  point  for  the  last  subtask  of  a  task  is  defined  as 
the  Closure  Call  time.  For  data  entry  challenges  (in 
general) ,  let  the  starting  time  be  the  exit  time  from  the 
previous  field,  and  let  the  stopping  time  be  the  time  of 
successful  completion  of  the  challenge  field.  For  menu 
selection  challenges,  let  the  starting  time  be  the 
processing  time  for  the  previous  subtask,  and  let  the 
stopping  time  be  the  time  of  correct  menu  selection  (when  it 
is  then  followed  immediately  by  continuation  of  subtask 
performance) .  Differences  in  Help  and  Non-Help  versions  of 
subtasks  and  challenges  was  calculated  and  t-tests  performed 
for  diagnostic  purposes. 

4.3.2  User  Satisfaction 

User  Satisfaction  was  determined  by  calculating  the 
mean  and  standard  deviation  of  user  responses  to  a 
comparative  question  on  the  Post-Test  Questionnaire 
(Question  #3)  as  discussed  in  Section  4.1.3. 

The  data  collected  in  the  Post-Task  Questionnaires  was 
quite  messy.  Many  user  responses  were  inconsistent  (as  when 
users  gave  opinions  on  messages  they  did  not  access)  and 
incomplete.  As  a  result,  the  data  on  user  opinions  of  how 
helpful  Dynamic  Help  was  when  used  may  not  be  informative. 

The  responses  collected  by  the  diagnostic  questions  on 
the  Post-Test  Questionnaire  appear  informative  and  will  be 
used  to  identify  and  rank  the  importance  of  each  structural 
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part  of  both  the  data  entry  and  menu  selection  screen 
Dynamic  Help  messages. 
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CHAPTER  V 


RESULTS,  CONCLUSIONS  AND  RECOMMENDATIONS 

5.1  Results 

This  chapter  presents  evaluative  and  diagnostic  results 
and  conclusions  of  the  experiment,  recommendations  for 
improvements  to  the  implementation  of  Dynamic  Help  in  ACIFS, 
and  suggestions  for  further  research  in  user  testing  of 
Embedded  User  Support  (EUS) . 

5.1.1  Evaluative  Results 

Table  5.1  summarizes  the  main  experimental  results  for 
evaluative  measures  (Dead  Ends,  errors,  time,  and  user 
satisfaction) ,  and  frequency  of  Dynamic  Help  access  per 
user.  These  results  aggregate  differences  between  Non-Help 
and  Help  versions  of  each  task. 

A  particular  user's  data  is  included  in  the  main  result 
of  a  listed  variable  only  if  complete  data  (every  version  of 
every  task)  was  present  for  that  user.  In  Table  5.1  (and 
many  other  tables  in  this  chapter) ,  N  is  the  number  of  users 
for  which  complete  data  was  present.  The  Mean  and  Standard 
Deviation  are  defined  as  usual  for  sample  statistics.  Let 
to  be  defined  as  the  calculated  paired  t-test  statistic  and 
let  P  be  defined  as  the  a  level  of  statistical  significance 
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Table  5.1  Main  Evaluative  Results  of  the  Experiment 


Measure 

M 

Mean 

Std  Dev 

ft 

0 

_ 

P 

Time  Difference 

15 

-470. 13 

1759 .25 

-1.03 

0.84 

Error  Difference 

9 

1.556 

2 . 08 

0.035 

Dead-End  Difference 

22 

3.591 

3.261 

5.16 

0.00 

Avg  #  Dyn  Hip  calls 

11 

18.727 

9.7272 

- 

- 

User  Satisfaction 

24 

1.25 

0.44233 

- 

- 

for  the  result  of  the  hypothesis  test  (Hq:  /1d  =  0  vs  Hj;  Md  > 
0,  where  =  Mean  Difference  (Without  Dynamic  Help  minus 
With  Dynamic  Help)).  Considering  a  =  0.05  to  be  a  standard 
threshold  of  significance  and  a  =  0.10  to  be  a  threshold  of 
suggestive  data,  the  author  will  report  P  and  will 
characterize  the  significance  or  suggestiveness  of  results 
by  comparing  P  or  1  -  P  to  0.05  or  0.10,  respectively. 
Interpretation  of  results  is  deferred  to  Section  5.2 
(Conclusions) . 

5. 1.1.1  Evaluative  Results  for  Performance  Time. 

Table  5.1  shows  that  users  performed  on  average  470.133 
seconds  more  slowly  with  Dynamic  Help  than  without  Dynamic 
Help  for  an  aggregated  set  of  15  users.  The  alternative 
hypothesis  was  that  Dynamic  Help  would  speed  up  performance 
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time.  Reversing  the  alternative  hypothesis  (H„:  Md  =  0  vs 
Hi:  Md  <  0)  consider  the  possibility  of  a  slowdown,  the 
average  difference  in  performance  time  would  be  significant 
for  a  levels  greater  than  0.16.  Thus  the  results  are  not 
significant  at  a  =  0.05  or  suggestive  at  a  =  o.io,  but  a 
Dynamic  Help  slowdown  is  implied  by  the  data.  Virtually  no 
user  support  time  was  added  for  each  Dead  End.  However,  if 
a  penalty  of  8.5  minutes  had  been  added  for  each  Dead  End, 
then  under  the  original  alternative  hypothesis.  Dynamic  Help 
would  have  saved  an  average  of  19.37  minutes,  suggestive  at 
the  a  s=  0.10  level,  and  with  a  penalty  of  12.5  minutes  added 
for  each  Dead  End,  Dynamic  Help  would  have  saved  an  average 
of  32.17  minutes,  significant  at  the  a  -  0.05  level. 

5. 1.1. 2  Evaluative  Results  for  User  Errors.  Tables 
5.1  and  5.2  show  that  Dynamic  Help  prevented  errors  for  a 
subset  of  nine  users.  But  Table  5.3  shows  that  Dynamic  Help 
caused  errors  for  the  entire  user  population  based  upon  the 
data  available  by  task.  Complete  error-count  data  for  the 
entire  test  was  present  for  only  nine  users,  including  five 
WANG  users  and  the  four  Non-WANG  users.  Many  user  results 
were  incomplete  either  because  of  missing  task  history  table 
cata  or  because  users  failed  to  properly  perform  or  complete 
the  task.  For  this  subset  of  nine  users,  the  average  number 
of  errors  per  user  for  four  tasks  was  5.555  with  Dynamic 
Help  4nd  7.11  without,  so  that  an  average  of  1.556  errors 
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Table  5.2  Error  Results  per  Task  for  the  Subset  of  Nine 

Complete-Error-Data  Users 


TASK 

#  Errors 

#  Errors 

#  Errors 

#  Errors 

with  Dyn 

without  Dyn 

prevented  by 

prevented 

Help 

Help 

Dyn  Help 

per  user 

11 

14 

24 

10 

1.1111 

12 

11 

13 

2 

0.2222 

13 

12 

16 

4 

0.4444 

15 

13 

11 

-2 

-0.2222 

Avg  per 

#  Errors 

usr  for 

5.555 

7.111 

prevented  per 

0.3888 

test 

user  per  task 
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were  "prevented"  by  Dynamic  Help  (see  Table  5.2).  This  21.9 
percent  error  reduction  would  be  significant  at  an  a  level 
of  at  least  0.035,  and  thus  is  significant  at  a  =  0.05. 

This  result  may  be  biased  to  represent  those  users  who 
carefully  followed  the  task  instructions  and  therefore 
generated  the  complete  data  necessary  for  inclusion  in  the 
result. 

Table  5.3  summarizes  the  error  results  by  task  for  all 
users  where  complete  data  was  available  for  both  versions  of 
a  task.  Errors  were  not  significantly  reduced  for  any  task. 
All  22  valid  users  are  represented;  there  was  complete  data 
on  at  least  one  task  for  all  of  them,  and  on  an  average  of 
3.5  out  of  4  tasks  for  them  as  a  whole.  In  fact.  Dynamic 
Help  "caused"  an  average  of  0.273  errors  per  user  per  task 
(an  increase  from  1.767  without  Dynamic  Help  to  2.040  with 
Dynamic  Help.  In  Task  15  there  was  a  suggestive  increase  in 
errors  with  Dynamic  Help  at  a  =  0.10  (t^  =  -1.51  and  P  = 
0.07).  These  mixed  results  (subset  of  nine  vs.  entire 
population  of  users)  will  be  interpreted  in  Section  5.2. 

5. 1.1. 3  Evaluative  Results  for  Dead  Ends.  There  were 
22  users  for  whom  complete  Dead  End  data  was  collected. 

With  Dynamic  Help,  the  number  of  Dead  Ends  per  user  for  four 
tasks  was  reduced  to  2.091  from  5.682  without  Dynamic  Help. 
Thus,  Dead  Ends  were  "prevented"  by  Dynamic  Help  an  average 
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Table  5.3  Error  Results  per  Task  for  Entire  User  Population 


TASK 

#  usrs  w/ 

complete 

data 

#  Errors 

w/Dyn 

Hip 

#  Errors 

w/o  Dyn 

Hip 

#  Errors 

Dyn  Help 

prevented 

#  Errors 

prevented 

per  user 

11 

21 

52 

50 

-2 

-0,095 

12 

21 

31 

31 

0 

0 

13 

18 

26 

25 

-1 

-0.056 

15 

17 

47 

31 

-16 

-0.941 

- 

- 

- 

- 

#  Errors 

prevented 

per  user 

per  task 

-0.273 
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Table  5.4  Dead  Ends  Results  per  Task 


TASK 

1191"  W/ 

complete 

data 

iDeadEnds 

w/Dyn  Hip 

#Dead 

Ends  w/o 

Dyn  Hip 

#DeadEnds 

Dyn  Hip 

prevented 

#DeadEnds 

prevented 

per  user 

11 

22 

5 

45 

40 

1.818 

12 

22 

12 

32 

20 

0.909 

13 

22 

10 

17 

7 

0.318 

15 

22 

19 

31 

12 

0.545 

- 

- 

- 

#DeadEnds 

prevented 

per  user 

per  task 

0.898 

of  3.591  times  per  user  during  the  four-task  testing 
session.  This  is  a  highly  significant  result  having  t^  = 
5.16  and  P  value  near  zero. 

Table  5.4  summarizes  the  Dead  End  results  by  task  for 
all  users  where  complete  data  was  available  for  both 
versions  of  a  task.  Dynamic  Help  prevented  an  average  of 
0.898  Dead  Ends  per  user  per  task. 

5. 1.1. 4  User  Satisfaction  Results.  Compiling  the 
responses  from  Question  #3  of  the  Post-Test  Questionnaire 
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yields  a  mean  response  of  1.25  (1  =  Very  Helpful,  2  = 
Somewhat  Helpful,  3  =  No  Difference,  4  =  Somewhat  Unhelpful, 
and  5  =  Very  Unhelpful)  as  the  users'  average  opinion  of 
whether  or  not  Dynamic  Help  was  helpful  while  using  the 
ACIFS  program.  Of  24  users,  18  of  them  gave  the  most 
favorable  response  on  the  5-point  response  scale. 

5. 1.1. 5  Use  of  Dynamic  Help.  When  available.  Dynamic 
Help  was  accessed  18.727  times  per  user  on  average  during 
the  testing  session,  or  4.682  times  on  average  per  task. 

When  at  a  particular  data  entry  field,  users  spent  an 
average  total  of  43.623  seconds  reading  the  Dynamic  Help 
message  and  completing  the  field  entry.  The  author  suspects 
that  some  of  the  Dynamic  Help  accesses  may  have  resulted 
from  curiosity  or  an  effort  to  please  the  Test 
Administrator . 

5.1.2  Diagnostic  Results 

5. 1.2.1  Diagnostic  Results  for  Performance  Time. 

During  testing,  the  author  observed  the  presence  of  three 
test  condition  effects  which  may  have  hampered  Dynamic 
Help's  ability  to  reduce  performance  time.  These  test 
condition  effects  are  discussed  in  Section  5. 2. 2.1. 

Locations  Where  Dynamic  Help  Saved  Time.  Table  5.5 
shows  the  one  location  identified  by  the  author  where 
Dynamic  Help  saved  performance  time  (measured  in  seconds) . 
This  same  challenge  (where  "6875"  was  the  only  acceptable 
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entry  for  a  six-and-seven-eighths  hat  size)  also  resulted  in 
the  most  significant  prevention  of  Dead  Ends.  This 
challenge  was  particularly  difficult  because  it  was 
virtually  impossible  for  the  user  to  guess  an  entry  which 
would  be  accepted  by  the  computer.  The  fact  that  Dynamic 
Help  was  so  time-saving  (over  half  a  minute  on  average)  is 
not  surprising. 


Table  5.5  Location  Where  Time  was  Saved 


Location 

N 

Mean 

Std  Dev 

B 

P 

Taskll/Size  K20163 

19 

32.895 

99.761 

1 . 44 

0.084 

Locations  Where  Dynamic  Help  Prolonged  Time.  Table  5.6 
lists  the  locations  identified  by  the  author  where  Dynamic 
Help  caused  an  increase  in  performance  time  (reversed 
alternative  hypothesis) .  In  every  case,  the  time  that  the 
user  spent  reading  Dynamic  Help  was  much  longer  than  the 
time  the  user  would  have  spent  in  simple  trial  and  error 
when  Help  was  not  available.  In  many  cases,  the  information 
that  a  user  searched  for  in  a  Dynamic  Help  message  was 
located  at  the  message's  end,  and  many  users  would  read  the 
entire  message  before  reaching  the  desired  information. 
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Ta^sle  5.6  Locations  Where  Time  was  Prolonged 


Location 

N 

Mean 

Std  Dev 

to 

p 

TASK  15 

18 

-388.67 

1096.11 

-1.50 

0.08 

Taskl2/Subtask  2 

H 

-49.238 

156.471 

Bi 

0.08 

Taskia/Subtask  3 

17 

-129.12 

300.01 

m 

0 . 05 

T12/Siibl/ESC-CondCd 

18 

-19.278 

62 . 566 

m 

0. 10 

T13/Sub3/MenuSelect 

Local  Cancellation 

16 

-104.19 

295.86 

-1.41 

0.09 

T15/Sub2/DOQTy 

16 

-62.625 

142.29 

m 

0.05 

Task  15  had  a  large  increase  in  time  that  appears  to 
have  resulted  from  Dynamic  Help  use.  A  difficulty  that 
users  had  which  was  not  anticipated  prior  to  testing  was  the 
menu  selection  for  Subtask  1  (complete  a  soldier's  initial 
issue) .  Although  data  was  not  collected  on  Dynamic  Help  use 
at  the  menu  screens,  the  author  observed  users  spending 
several  minutes  in  Dynamic  Help  message  screens  attempting 
to  determine  the  correct  menu  selection  to  begin  Subtask  1, 
and  in  many  cases  users  declared  Dead  Ends.  The  DOQTY 
challenge  (also  listed  in  Table  5.6)  was  significant  against 
Dynamic  Help  for  time.  This  contributed  to  the  suggestive 
slowdown  indicated  by  Task  15 ‘s  time  difference.  The 
Dynamic  Help  message  for  the  DOQTY  challenge  appeared  to  be 
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difficult  to  read  for  most  users  accessing  it,  and  therefore 
the  time  spent  reading  the  message  probably  caused  the 
significant  difference  in  performance  time  for  that 
challenge. 

In  Task  13  Subtask  3  (Post  a  Local  Cancellation) ,  the 
difference  in  time  was  significant  mainly  because  of  the 
time  it  took  to  use  Dynamic  Help  in  the  proper  menu 
selection  to  begin  the  subtask  (see  T13/Sub3/MenuSelect  Post 
Local  Cancellation  in  Table  5.6). 

5. 1.2.2  Diagnostic  Results  for  User  Errors.  Both  with 
and  without  Dynamic  Help,  the  nine  Complete-Error-Data  users 
(those  for  whom  complete  error  data  could  be  collected  on 
all  tasks)  committed  114  errors  in  72  task  sessions,  or 
1.583  errors  per  user  per  task.  The  other  users  committed 
179  errors  in  82  task  sessions,  or  2.183  errors  per  user  per 
task.  Both  these  error  rates  are  far  greater  than  would  be 
tolerated  in  an  actual  job  setting.  Throughout  testing,  the 
author  observed  that  users  were  not  motivated  to  use  Dynamic 
Help  for  error  prevention;  they  appeared  "careless"  and  not 
as  quality-conscious  as  expected.  Although  the  tasks  were 
designed  to  be  as  challenging  as  possible  (and  hence  more 
error-prone  than  other  typical  CIF  tasks) ,  it  appears  that 
users  tolerated  a  much  higher  error  rate  than  would  have 
been  permitted  in  their  actual  job  settings.  The  author 
suspects  that  the  Complete-Error-Data  users'  behavior  was 
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closer  to  that  in  a  realistic  job  setting  than  the  behavior 
of  other  users.  In  the  remainder  of  this  section,  the 
author  will  analyze  Dynamic  Help's  prevention  and  causing  of 
errors  with  Dynamic  Help  and  then  will  analyze  Dynamic 
Help's  prevention  of  errors  for  the  nine  Complete-Error-Data 
users . 

Locations  Where  Dynamic  Help  Prevented  Errors  for  the 
Entire  User  Population.  Table  5.7  lists  the  single 
identified  location  where  Dynamic  Help  prevented  errors  on 
the  whole.  Subtask  2  of  Task  12  is  the  Statement  of  Charges 
creation  and  correction  and  will  be  discussed  in  Section 
5 . 2 . 2 . 2  . 


TzQ^le  5.7  Location  Where  Errors  were  Prevented  for  the 

Entire  User  Population 


Location 

N 

Mean 

Std  Dev 

B 

P 

Taskl2/Subtask  2 

21 

0.429 

1.207 

1.63 

0.06 

Locations  Where  Dynamic  Help  Caused  Errors  for  the 
Entire  User  Population.  Table  5.8  shows  those  identified 
places  in  the  ACIFS-modif ied  program  where  a  reversed 
alternative  hypothesis  test  (Hq:  Md  ==  0  vs  HjI  fJ-j)  <  0)  showed 
Dynamic  Help  as  causing  errors  for  the  entire  user 
population. 
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Table  5.8  Locations  Where  Errors  were  Caused  for  the  Entire 


User  Population 


Location 

N 

Mean 

Std  Dev 

ft 

0 

_ ^ 

p 

TASK  15 

17 

-0.412 

1.121 

-1.51 

0.07 

Taskll/PrntOut  0th  Errs 

22 

-0.273 

0.827 

-1.55 

0.07 

Taskl2/8ubtask  1 

H 

-0.381 

0.921 

-1 . 90 

0 . 04 

Taskl3/8ubtask  2 

18 

-0.222 

0.548 

Bl 

0.05 

T12/8ubl/Condition  Code 

20 

-0.2 

0.696 

m 

0.11 

T15/SUb2/DOQTy 

18 

-0.111 

0.3234 

m 

0.08 

For  Task  15,  most  of  the  errors  were  made  as  "Other" 
errors  in  Subtask  1  or  as  errors  in  Subtask  2.  Neither  of 
these  types  of  errors  was  significant  by  itself  but  in 
combination  they  force  the  data  on  errors  to  be  suggestive 
that  Dynamic  Help  caused  errors  in  the  task.  The  "Other" 
errors  in  Subtask  1  were  non-challenge  errors  and  were 
probably  lack-of-attention-to-detail  errors  while  errors 
made  in  Subtask  2  were  mostly  challenge-related  errors  which 
may  have  been  aggravated  by  poorly  written  Dynamic  Help 
messages.  In  particular,  the  DOQTY  challenge  error  results 
are  significant  against  Dynamic  Help  and  could  very  well 
have  been  caused  by  a  poorly  written  or  poorly  structured 
Dynamic  Help  message.  To  identify  how  "bad"  the  Help 
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message  was,  the  author  determined  whether  the  error  results 
were  still  significant  for  those  users  who  actually  accessed 
Dynamic  Help,  and  found  that  the  results  for  users  actually 
using  Help  remained  against  Dynamic  Help  (t^  =  -1.51  and  P  = 
0.08).  This  suggests  that  the  help  message  had  an  adverse 
impact  on  error  rates  for  that  challenge. 

The  Task  11  "Printout-Other"  (Non-Challenge)  error 
results  suggest  that  Dynamic  Help  caused  errors  and  could 
have  been  caused  by  poorly  written  or  incomplete  Dynamic 
Help  messages. 

In  Task  12  Subtask  1,  errors  occurred  in  the  posting  of 
NSN  quantities  or,  more  often,  errors  occurred  in  the  input 
of  the  Condition  Code.  The  incorrect  postings  of  quantities 
were  most  likely  caused  by  a  failure  to  read  the  task 
documentation  as  opposed  to  any  failure  of  Dynamic  Help 
messages.  According  to  the  table,  Condition  Code  errors 
were  also  significant  against  Dynamic  Help  at  a  levels 
greater  than  or  equal  to  0.11.  But,  in  attempting  to 
analyze  how  deficient  the  Condition  Code  Dynamic  Help 
message  was  (whether  or  not  it  caused  confusion  which  caused 
errors) ,  the  author  found  that  in  those  cases  where  Help  was 
actually  accessed  by  the  user,  errors  were  not  significant 
(to  =  0.0  and  P  =  0.50).  Thus,  Dynamic  Help  did  not  cause 
the  error  significance  in  Task  12  Subtask  1  and  in  the 
Condition  Code  challenge. 
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In  Task  13  Subtask  2  (create  a  supply  requisition  from 
an  unreadable  NSN)  there  was  one  potential  challenge 
(correct  the  NSN) .  No  errors  were  committed  in  the 
encounters  of  that  particular  challenge.  Most  of  the  errors 
users  committed  in  this  subtask  were  caused  by  a  failure  to 
follow  task  instructions.  The  author  feels  that  the 
significance  in  errors  for  this  subtask  is  not  due  to 
Dynamic  Help. 

Locations  Where  Errors  were  Prevented  for  the  Nine 
Complete-Error-Data  Users.  Table  5.9  shows  those  identified 
locations  where  Dynamic  Help  prevented  errors  for  the  nine 
Complete-Error-Data  users.  In  contrast  to  the  corresponding 
results  for  the  entire  user  population,  these  results  are  as 
expected,  with  statistically  significant  or  suggestive  error 
prevention  at  five  locations  that  were  challenging. 

Task  11  errors  overall,  the  Task  11  ''Printout- 
Challenge”  errors,  and  the  Task  11  Size  of  D01857  and  C08256 
were  significantly  prevented  by  Dynamic  Help.  This  is  not 
unusual  since  Task  11  had  several  difficult  Size  challenges 
as  well  as  other  data  input  challenges.  Without  Dynamic 
Help,  Complete-Error-Data  users  could  be  expected  to  use 
trial  and  error  to  complete  some  entries,  resulting  in 
incorrect  yet  range-acceptable  guesses. 

Subtask  3  of  Task  12  (Modify  a  Report  of  Survey)  was 
challenging  to  users  for  three  reasons,  two  of  which  were 
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error-causing.  First,  users  did  not  know  what  NSN  and 
quantity  to  enter  on  the  worksheet.  As  a  result,  those 
without  access  to  Dynamic  Help  could  not  get  an  explanation 
of  worksheet  completion  procedures.  Most  users  would  then 
guess  anj  enter  incorrect  data  which  were  within  the  range 
of  acceptable  entries.  Second,  users  were  confused  about 
which  datum  to  enter  in  the  Total  Price  field  of  the 
worksheet  and  many  would  enter  the  wrong  price.  The  error 
rate  decreased  with  Dynamic  Help  for  the  first  challenge  and 
increased  for  the  second.  Therefore  the  Dynamic  Help 
message  for  the  worksheet  completion  procedures  appears  to 
be  helpful,  but  the  Dynamic  Help  message  for  the  Total  Price 
field  may  be  unclear  or  need  restructuring. 

Table  5.9  Locations  Where  Errors  were  Prevented  for  the 


Nine  Complete-Error-Data  Users 


Location 

N 

Mean 

Std 

B 

P 

TASK  11 

9 

1.111 

1.269 

2.63 

0.015 

Taskll/PrntOut  Chal  Errs 

9 

1.222 

1.093 

3.35 

0.005 

Tas]ci2/Subtaslc  3 

9 

0.222 

0.441 

1.51 

0.085 

Taslcl3/Subtask  3 

9 

0.222 

0.441 

1.51 

0.085 

Tll/Size  D01857  &  C08256 

9 

0.444 

0.527 

2.53 

0.018 
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In  Subtask  3  of  Task  13  users  were  instructed  to 
perform  a  local  cancellation  on  a  supply  requisition,  and 
the  only  error-causing  confusion  in  the  associated  worksheet 
was  the  Quantity  to  Cancel  field.  Many  users  without 
Dynamic  Help  were  confused  about  what  quantity  to  enter  in 
the  field  and  subsequently  made  incorrect  quesses  which  were 
range-acceptable.  Help  users  were  able  to  access  needed 
information,  and  thus  were  assisted  in  correctly  completing 
the  entry. 

In  contrast  to  the  situation  for  the  entire  user 
population  in  which  Dynamic  Help  seemed  to  cause  errors 
(Table  5.8),  for  the  Complete-Error-Data  users  there  were  no 
identifiable  locations  where  there  were  significantly  more 
errors  with  Dynamic  Help  than  without. 

5. 1.2. 3  Diagnostic  Results  for  Dead  Ends.  Locations 
Where  Dynamic  Help  Prevented  Dead  Ends.  As  shown  in  Table 
5.10,  the  author  identified  3  task  types,  4  subtasks,  and  9 
challenge  points  where  Dynamic  Help  prevented  Dead  Ends. 

Task  11  (Manual  Turn  In) ,  along  with  its  Size  challenge  for 
LIN  Number  K20163,  had  highly  significant  results  (P  value 
is  near  zero) .  Out  of  the  nine  significant  challenges,  five 
of  them  were  Size  input  challenges  where  the  user  would  not 
be  likely  to  know  the  correct  size  format.  In  these  cases. 
Dynamic  Help  provided  the  user  with  a  list  of  correct  size 
formats  to  choose  from.  The  Task  12/Subtask  1/Condition 
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Teible  5.10  Locations  Where  Dead  Ends  were  Prevented 


Location 

N 

Mean 

Std  Dev 

B 

P 

TASK  11 

22 

1.818 

1 . 402 

6.08 

0.0 

TASK  12 

22 

0.909 

1 . 306 

3 .26 

0,002 

TASK  15 

22 

0.54  5 

1.945 

1.32 

0.10 

Task  12/Subtask  1 

22 

0.455 

0.739 

2.89 

0.004 

Task  12/Subtask  2 

22 

0.1364 

0.4676 

1.37 

0,093 

Task  12/Subtask  3 

22 

0.318 

0.716 

2.08 

0.025 

Task  13/Subtask  1 

22 

0.227 

0.752 

1.42 

0,085 

Taskll/sizeC07440 

18 

0.167 

0.514 

1.37 

0.094 

Taskll/SizeK20l63 

22 

0.7727 

0.4289 

8.45 

0.00 

Taskll/SizeD01857&C08256 

9 

0.222 

0.441 

1.51 

0.085 

T12/Subl/ESC  to  cond  Cd 

20 

0.10 

0.3078 

1.45 

0.081 

Tl2/6\ibl/ConditionCode 

21 

0.333 

0.483 

3.16 

0.003 

T12/8ub2/ESC  to  TotPrice 

21 

0.1905 

0.4024 

2.17 

0.021 

TasklS/Subl/Size  C07440 

21 

0.286 

0.561 

2.34 

0.015 

Taskl5/8ubl/8iZ6  N39648 

22 

0.2273 

0.4289 

B 

0.011 

Taskl5/8ub2/Menu8elect 

of  Issue  Due-Out  Items 

22 

0.1364 

0.4676 

1.37 

0.093 

Code  challenge  is  another  test  of  user  recall  and 
recognition,  and  Dynamic  Help  provided  the  user  with  a  list 
of  choices  and  descriptions  so  that  a  proper  selection  could 
be  determined.  Two  of  the  other  challenges  deal  with  the 
use  of  the  ESC  Key  to  move  to  another  portion  of  the  screen. 
The  global  portion  of  the  Dynamic  Help  message  laid  out  the 
procedure  for  moving  within  the  screen,  allowing  the  user 
enough  information  to  continue  with  the  task. 

There  are  two  table  entries  whose  underlying  reasons 
for  Dead  End  prevention  may  not  be  apparent.  They  are  the 
Subtask  3  of  Task  12  and  Subtask  1  of  Task  13  entries. 
Subtask  3  of  Task  12  may  be  significant  because  of  two 
challenges:  first,  the  proper  menu  selection  to  begin  the 

subtask  (whose  significance  level  is  a  -  0.16)  and  second, 
the  total  price  to  cancel  on  the  Report  of  Survey 
modification  (whose  significance  level  is  a  =  0.16).  The 
combination  of  these  two  challenges  provided  statistical 
significance  to  the  entire  subtask.  Subtask  1  of  Task  13 
involved  the  creation  of  supply  requisitions,  and  the  user 
may  have  been  confused  at  many  points  about  proper  code 
entries  when,  in  fact,  many  of  those  entries  were  optional. 
Without  Dynamic  Help,  many  users  were  unable  to  determine 
whether  or  not  the  code  entries  were  optional  in  some 
fields,  and  they  declared  Dead  Ends. 
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Locations  Where  Dynamic  Help  Caused  Dead  Ends.  The 
author  could  identify  no  specific  places  where  Dynamic  Help 
actually  caused  a  Dead  End,  although  there  were  some  places 
in  the  program  where  Dynamic  Help  was  not  clear  enough,  so 
that  users  declared  Dead  Ends,  but  not  to  a  greater  extent 
than  without  Dynamic  Help. 

5. 1.2. 4  Other  Diagnostic  Performance  Results.  Through 
analysis  and  personal  observation,  the  author  discovered 
several  other  indicators  of  the  usefulness  of  Dynamic  Help 
to  novice  ACIFS  users. 

When  available.  Dynamic  Help  was  an  alternative  to 
trial-and-error  data  entry.  A  successful  Dynamic  Help 
access  would  presumably  prevent  further  incorrect  trials,  so 
it  was  hypothesized  that  Dynamic  help  should  reduce  the 
number  of  incorrect  guesses  prior  to  a  final  data  entry. 

The  author  determined  the  number  of  incorrect  guesses  or 
trials  until  the  completion  entry  for  certain  challenge 
fields  and  analyzed  whether  the  difference  in  the  number  of 
wrong  guesses  would  be  significant  between  the  Non-Help  and 
Help  versions  of  the  program.  Table  5.11  shows  that  four  of 
the  data  entry  challenge  results  were  significant  or 
suggestive  in  the  difference  in  number  of  wrong  trials  or 
guesses  prior  to  the  final  entry.  Three  of  the  four  were 
Size  challenges  and  the  other  one  was  the  Taskl2/Subtaskl/ 
ESCtoCondCode  challenge.  These  results  may  indicate  that 
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Table  5.11  Locations  Where  Dynamic  Help  Prevented  Wrong 


Guesses 


Location 

N 

Mean 

Btd  Dev 

B 

P 

Taskll/Size  K20163 

20 

3.50 

4 . 371 

3 . 58 

0.0 

T12/Subl/ESCtoCondCd 

18 

0.778 

1.957 

1.69 

0.055 

T12/Sub2/ESCtoTotPrice 

20 

2.40 

8.513 

0.11 

T15/8ubl/8ize  C07440 

17 

1.294 

3.424 

1.56 

0.069 

T15/8ubl/8ize  N39848 

- 1 

1.176 

2 . 628 

1.85 

0.042 

Dynamic  Help  decreased  the  number  of  incorrect  trials  before 
entry  completion  for  certain  size  fields  and  movement  within 
screen  actions. 

If  a  user  made  incorrect  guesses  of  a  data  entry  after 
accessing  Dynamic  Help,  this  could  indicate  deficiencies  in 
the  help  message.  For  instance,  although  the  difference  in 
wrong  guesses  overall  for  the  T12/Sub2/ESCtoTotPrice 
challenge  is  in  favor  of  Dynamic  Help  (t^  =  1.26  and  P  = 
0.11),  users  made  an  average  of  4.8571  wrong  guesses  after 
calling  Dynamic  Help  for  the  first  time.  This  may  indicate 
that  Dynamic  Help  was  useful  but  that  the  message  could  be 
made  clearer  or  more  complete  to  facilitate  better 
performance . 
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The  author  observed  that  roost  users  had  difficulty  in 
selecting  the  proper  roenu  item  from  the  Main  Menu  to  start  a 
task.  These  initial  menu  selection  problems  were  not 
anticipated  by  the  author  and  seemed  to  be  experienced  most 
by  users  who  were  familiar  with  the  WANG  computer  program. 
WANG  users  often  would  misinterpret  the  ACIFS  terminology 
similar  to  that  of  their  system.  These  misunderstandings 
were  especially  prevalent  in  the  Main  Menu  and  Submenu 
screens.  This  was  substantiated  by  WANG  user  comments.  The 
author  also  noted  that  the  users  from  Ft.  Polk  who  use  a 
local  computer  program  had  no  difficulty  with  menu  selection 
in  Subtask  2  of  Task  15  (Issue  Due-Out  Items)  where  the  WANG 
users  were  consistently  challenged.  The  Ft.  Polk  CIF 
program  may  have  been  similar  in  terminology  for  that 
particular  subtask  menu  selection.  There  were  other 
performance  errors  that  users  made  because  of  WANG 
experience,  such  as  in  the  posting  of  turn-in  quantities. 
WANG  users  normally  post  the  clothing  record  with  zeros 
(showing  that  soldier  is  no  longer  signed  for  any  clothing) 
while  the  ACIFS  program  requires  posting  of  actual  turn-in 
quantities.  But  overall,  most  of  the  confusions  and  errors 
resulting  from  familiarity  with  another  CIF  program  were 
observed  in  menu  selection  processes. 

Although  Dynamic  Help  was  used  in  menu  selection 
screens,  users  tended  to  use  trial  and  error  as  their 
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primary  means  of  determining  correct  menu  choices.  Users 
may  have  felt  that  it  was  much  easier  and  quicker  to  simplj 
try  all  of  the  few  promising  screens  (2  to  5)  before 
resorting  to  use  of  Dynamic  Help. 

The  author  also  noted  that  there  were  several 
preidentified  challenge  points  where  user  efficiency  and 
accuracy  could  have  been  increased  (in  correction 
procedures,  decision-making,  or  in  determining  a  worksheet's 
quickest  completion  method)  with  the  use  of  Dynamic  Help. 
However,  most  users  failed  to  use  help  because  they  did  not 
realize  that  a  challenge  to  their  efficiency  or  accuracy  was 
present  and  as  a  result.  Dynamic  Help  was  not  found  to  be 
significantly  helpful.  Some  of  these  potential  challenges 
include  the  Special  Issue  question  in  Task  11.  The  required 
user  action  was  to  respond  either  "Y"  or  "N”  to  a  query 
about  the  soldier's  MOS.  Most  users  made  guesses  based  on 
experience,  declared  a  Dead  End  very  early,  or  answered  "N" 
(because  the  attached  paperwork  did  not  contain  the 
information)  not  realizing  that  information  with  which  to 
make  the  correct  decision  was  present  in  the  Dynamic  Help 
message.  Another  potential  challenge  was  the  method  of 
quickest  movement  through  the  Complete-the-Issue  Worksheet 
(Task  15/Subtask  1) .  Even  though  users  were  explicitly 
instructed  to  move  through  the  worksheet  as  quickly  as 
possible,  most  did  not  access  Dynamic  Help  for  information 
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on  quickest  movement.  Many  of  these  unrealized  potential 
challenges  were  not  data-entry,  field-specific  items.  Most 
were  global  informational  items.  In  all  of  these  cases, 
because  the  user  failed  to  recognize  that  there  was  a  need 
for  information,  Dynamic  Help  was  not  accessed  and  therefore 
could  not  be  useful. 

5. 1.2. 5  Diagnostic  User  Opinion  Results.  In  Question 
#2  of  the  Post-Test  Questionnaire  users  were  asked  to  rank, 
in  order  of  importance,  the  three  reasons  why  they  might 
have  sought  Dynamic  Help  in  a  menu  screen,  with  a  ranking  of 
"1"  being  the  information  most  sought  and  "3"  being  the 
least.  Table  5.12  shows  that  users  sought  help  for  the 
Procedures  category  (how  to  make  a  menu  selection)  most 
often,  with  12  of  the  24  users  ranking  it  first.  They 
sought  help  to  Understand  the  Meanings  of  listed  menu 
selections  the  next  most  often  and  finally,  users  sought 
help  for  General  Orientation  the  least  often  with  11  of  the 
24  users  ranking  it  last. 

In  Question  #1  of  the  Post-Test  Questionnaire  users 
were  asked  to  rank,  in  order  of  importance,  the  five  reasons 
why  they  might  have  sought  Dynamic  Help  in  a  data-entry 
screen  with  a  ranking  of  "1”  being  the  information  most 
sought  and  ”5"  the  least.  Table  5.13  shows  that  users 
clearly  felt  that  they  needed  help  most  often  for  the  What 
to  Enter  category  with  15  out  of  24  users  ranking  it  most 
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Tabic 


User  Help-Seeking  Preferences  at  Menus 


Category 

N 

Mean 

Std  Dev 

Procedures 

24 

1,667 

0 . 7614 

Understanding  Meanings 

Bi 

2 . 083 

0.8297 

General  Orientation 

24 

2 .25 

0.794 

Table  5.13  User  Help-Seeking  Preferences  at  Data-Entry 

Screens 


Category 

M 

Mean 

Std  Dev 

What  to  Enter 

24 

1.542 

0 . 833 

How  to  Enter 

24 

2.2917 

1.1971 

Location 

24 

3 . 2083 

1.3507 

Movement  Within  Screens 

24 

3.75 

0 . 944 

Movement  Between  Screens 

24 

4.2083 

0.833 

101 


important  and  no  users  ranking  it  least  important.  This  is 
understandable  because  the  most  challenging  locations  in  the 
program  were  at  places  where  the  user  would  not  be  able  to 
determine  the  correct  format  of  a  data  entry.  The  How  to 
Enter  category  was  second,  with  5  out  of  24  users  ranking  it 
first  and  13  users  ranking  it  second.  Movement  Between 
Screens  was  the  category  sought  least  often,  which  is 
understandable  because  ACIFS  has  a  rigid  screen  sequence. 

5.2  Conclusions 
5.2.1  Evaluative  Conclusions 

Based  upon  the  analysis  and  results  discussed  above, 
the  author  concludes  that  Dynamic  Help  significantly  reduces 
novice  and  intermittent  user  Dead  Ends.  Also,  it 
significantly  reduces  errors  for  users  for  whom  complete 
error  data  was  available,  but  for  other  users  did  not  reduce 
errors  under  test  conditions.  Assuming  that  the  Complete- 
Error-Data  users  behaved  more  like  users  in  an  actual  job 
setting,  then  Dynamic  Help  reduces  errors.  Dynamic  Help  did 
not  improve  user  performance  times  under  test  conditions. 

In  fact,  the  results  suggest  that  overall  performance  time 
was  slightly  increased  when  Dynamic  Help  was  available  for 
use.  For  the  subset  of  nine  Complete-Error-Data  users,  the 
slowdown  was  slightly  greater,  and  the  same  conclusion 
applies  as  for  all  the  users. 
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An  analysis  of  the  Post-Test  Questionnaire  indicated 
that  users  seemed  satisfied  with  the  helpfulness  of  Dynamic 
Help. 

5.2.2  Diagnostic  Conclusions 

5.2.2. 1  Diagnostic  Conclusions  for  Time.  Table  5.14 
summarizes  the  possible  reasons  for  unfavorable  time  results 
with  Dynamic  Help.  As  discussed  in  Section  5. 1.2.1,  user 
performance  times  were  enhanced  by  Dynamic  Help  only  for 
those  data  entry  fields  where  the  user  was  required  to 
recall  difficult  data  entry  formats  from  memory.  In  those 
cases,  Dynamic  Help  saves  part  of  the  time  that  a  user  would 
spend  searching  their  memory  for  and  entering  as  many 
combinations  of  formats  as  necessary  until  the  program 
accepts  one.  Otherwise,  Dynamic  Help  does  not  significantly 
reduce  performance  time  and  may  somewhat  degrade  it.  This 
degradation  occurs  in  part  because  the  users  must  search  the 
help  text  for  the  information  they  are  seeking,  and  for  many 
challenge  points,  the  information  desired  was  located  at  the 
end  of  the  help  message.  This  indicates  that  Dynamic  Help 
messages  may  need  structural  changes  to  assist  users  in 
locating  the  information  they  are  most  likely  to  seek.  A 
restructuring  of  Dynamic  Help  messages  would  possibly  result 
in  a  reduction  of  user  performance  times. 
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Table  5.14  Possible  Mechanisms  Causing  Unfavorable  Time 

Results 


Classification 

Mechanism 

1.  Dynamic  Help 

*  Poor  message  structure 

Deficiencies 

*  Message  content  unclear 

2 .  Control  of 

*  Overuse  of  Help  due  to 

curiosity  &  desire  to  please 

Test  Conditions 

*  Poor  time  discipline 

*  High  user  tolerance  for  errors 

Another  reason  for  lengthy  help  message  reading  may  be 
that  users  cannot  quickly  interpret  the  information  located 
in  the  help  message  either  because  the  message  is  unclear  or 
the  information  is  not  clearly  translated.  For  instance,  in 
certain  size  fields,  such  as  that  for  LIN  C07440  in  Task  11, 
the  format  for  entry  is  highly  unusual  (5  1/2  Regular  = 

055R) ,  and  the  help  message  does  not  clearly  show  a  user  how 
to  decide  what  datum  to  enter.  Therefore,  it  takes  an 
unacceptable  amount  of  time  (approximately  82  seconds)  for 
users  to  choose  and  input  the  correct  size  because  there  is 
no  translation  or  description  of  how  to  determine  the 
correct  datum  from  the  listed  entries. 
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The  author  found  several  factors  degrading  Dynamic 
Help's  improvement  of  performance  time  that  were 
attributable  to  test  conditions  (see  Table  5.14).  First,  in 
an  actual  job  setting.  Dynamic  Help  would  have  been  accessed 
less  frequently;  curiosity  and  desire  to  please  the  Test 
Administrator  would  not  be  present.  A  decrease  in  non- 
essential  use  would  reduce  the  overall  degradation  of 
performance  time. 

Secondly,  time  was  disciplined  poorly  because  of  user 
calendar  requirements  and  the  possible  resultant  user 
interactions.  Users  for  any  given  test  day  would  have 
previously  made  flight  arrangements  for  the  late  afternoon, 
both  leaving  on  the  same  flight.  One  user  would  perform 
tasks  more  quickly  than  the  other  user.  Users  could  have 
applied  pressure  on  each  other  to  speed  up  or  slow  down. 
During  their  lunch  break,  both  users  could  have  agreed  on  a 
set  target  completion  time  to  ensure  they  had  ample  time  to 
make  their  flight.  The  slower  of  the  two  users  could  be  in 
a  hurry  in  the  afternoon  and  possibly  would  risk  more 
errors.  The  quick  user  could  slow  down,  and  possibly  would 
overuse  Dynamic  Help. 

All  of  this  is  consistent  with  the  much  faster 
performance  times  in  the  afternoon  sessions  (which  could 
also  be  due  to  a  simple  learning  effect) .  It  would  seem  to 
explain  the  greatly  increased  error  rate  in  the  afternoon. 
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especially  since  the  afternoon  error  rate  should  have  been 
decreased  by  the  learning  effect.  The  afternoon  rate  of 
2.1585  errors  per  user  per  task,  versus  1.8333  in  the 
morning,  strongly  suggests  carelessness  or  fatigue  in  the 
afternoon.  It  is  easy  to  imagine  that  lime-pressured  users 
would  avoid  investing  time  in  reading  help  messages,  while 
more  skilled,  non-pressured  users  would  browse.  This  would 
explain  why  the  test  showed  Dynamic  Help  as  costing  time 
rather  than  saving  time. 

The  final  test  condition  effect  on  the  time  results  is 
user  tolerance  for  error.  During  the  test  a  user  had  three 
choices  in  dealing  with  confusing  locations  in  the  program 
and  moving  on  with  a  task:  1)  spend  additional  time  making 
guesses,  searching  the  screen  for  instructions,  etc.,  2) 
declare  a  Dead  End  (in  lieu  of  user  support) ,  or  3)  make  an 
error  (with  or  without  an  intention  to  correct  it  later) . 
While  doing  the  same  task  in  an  actual  job  setting  the  user 
would  not  have  the  third  choice  (make  an  error) ,  but  would 
have  to  spend  additional  time  trying  to  resolve  the 
confusion  either  by  trial  and  error  or  by  accessing  user 
support.  The  author  observed  that  users  tended  to  have  a 
high  tolerance  for  errors  during  testing.  Th.LS  observation 
is  supported  by  the  fact  that  user  errors  were  greater  in 
the  afternoon  than  in  the  morning  (see  preceding  paragraph) , 
contrary  to  what  would  be  expected  from  a  learning  effect. 
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As  users  went  through  a  day  of  testing,  their  tolerance  for 
errors  seemed  to  increase.  As  user  error  tolerance 
increased,  the  time  spent  resolving  confusing  situations 
decreased.  This  had  an  unanticipated  impact  on  Dynamic 
Help's  ability  to  reduce  performance  times. 

Because  these  three  conditions  were  present  during 
testing,  the  author  concludes  that  the  testing  has  not  shown 
that  Dynamic  Help  would  fail  to  save  time  in  an  actual  job 
setting.  It  is  possible  that  in  a  better  test  or  in  an 
actual  job  setting.  Dynamic  Help  would  save  time. 

It  should  be  noted  that  in  the  current  implementation 
most  help  messages  were  retrieved  intact  from  files  rather 
than  assembled  by  query  at  run  time.  Thus,  message  access 
was  rapid,  and  the  author  observed  no  significant 
degradation  in  performance  time  due  to  message  retrievals. 
Retrievals  occurred  faster  than  user  reading  speed.  If  the 
building  of  help  messages  becomes  slow  in  future 
implementations,  performance  time  may  be  further  increased 
by  Dynamic  Help. 

5. 2. 2. 2  Diagnostic  Conclusions  for  Errors .  Diagnosti c 
Conclusions  for  Errors  for  the  Entire  User  Population.  The 
author  found  one  location  where  errors  were  significantly 
reduced  by  Dynamic  Help  for  the  entire  population  of  users, 
but  Dynamic  Help  was  not  the  primary  cause  for  the  reduction 
of  errors  in  that  case.  The  author  feels  that  there  are  two 
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areas  in  Subtask  2  of  Task  12  that  accounted  for  most  of  the 
errors.  The  first  is  the  user's  failure  to  properly  correct 
an  entry  in  the  Worksheet  (as  instructed  midway  through  the 
subtask).  The  second  is  the  user's  failure  to  enter  the 
correct  Total  Price.  Neither  of  these  were  defined  as 
potential  challenges.  The  most  likely  reason  why  users  made 
these  mistakes  is  a  failure  to  follow  instructions  and  read 
the  provided  documentation.  The  author  sees  no  mechanism  to 
ascribe  this  error  prevention  directly  to  the  availability 
and  use  of  Dynamic  Help. 

In  general,  the  entire  population  of  users  seemed  to 
commit  more  errors  with  Dynamic  Help  under  test  conditions 
as  opposed  to  the  number  of  errors  they  would  be  likely  to 
commit  in  an  actual  job  setting.  The  author  concludes  that 
there  are  two  main  reasons  for  the  increased  error  rate 
because  of  Dynamic  Help.  First,  users  may  become  more 
confident  when  they  have  access  to  help  and  therefore  are 
less  careful  in  their  actions,  and  second,  certain  Dynamic 
Help  messages  are  poorly  written  or  incomplete  and  users 
misinterpret  the  information  contained  in  the  message. 
However,  because  user  tolerance  for  errors  should  be  lower 
in  an  actual  job  setting  than  occurred  in  the  user  testing 
(see  Section  5. 2. 2.1  for  discussion  of  error  tolerance),  the 
author  would  expect  users  to  be  more  careful  and  that 
Dynamic  Help  would  be  accessed  more  often  specifically  for 
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error  prevention.  As  a  result,  it  is  hoped  that  errors 
would  be  significantly  reduced  by  the  Dynamic  Help 
implementation. 

Diagnostic  Conclusions  for  the  Nine  Complete-Error-Data 
Users.  The  results  for  the  subset  of  nine  users  discussed 
in  Section  5. 1.2.2  indicate  that  Dynamic  Help  prevented 
data-entry  as  well  as  worksheet  instruction  errors  and 
support  the  conclusion  that  Dynamic  Help  would  be  useful  in 
preventing  errors  for  Complete-Error-Data  users,  who  are 
assumed  to  closely  approximate  users  in  an  actual  job 
setting.  However,  some  help  messages  need  to  be  clarified 
or  restructured  (i.e.  Total  price  field  in  Modify  Report  of 
Survey) . 

5. 2. 2. 3  Diagnostic  Conclusions  for  Dead  Ends.  Because 
Dynamic  Help  prevents  Dead  Ends  so  significantly,  especially 
in  data  entry  fields  where  a  user  must  recall  the  correct 
format  or  the  correct  method  of  movement  (see  Table  5.9), 
the  author  concludes  that  Dynamic  Help  is  most  useful  in 
those  areas  where  the  computer  program  will  not  allow  the 
user  to  move  ahead  with  a  task  unless  the  data  entry  format 
is  correct,  and  the  user  is  unable  to  recall  the  correct 
format  entry  from  memory.  The  addition  of  Dynamic  Help 
allows  novice  and  intermittent  users  to  traverse  through 
these  difficult  areas,  assisting  completion  of  the  most 
common  tasks  with  almost  no  prior  training.  However, 
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Dynamic  Help  does  not  reduce  Dead  Ends  to  zero,  and  thus 
does  not  hold  the  promise  of  substituting  for  all  training, 
other  documentation,  and  user  assistance. 

5. 2. 2. 4  Other  Diagnostic  Conclusions  for  Performance . 
For  certain  size  fields  and  movement-within-screen  actions 
Dynamic  Help  decreases  the  number  of  wrong  guesses  prior  to 
completion  of  the  field  or  action.  However,  Dynamic  Help 
does  not  totally  eliminate  wrong  guesses  after  invocation  in 
certain  instances,  and  the  author  concludes  that  continued 
wrong  guessing  is  due  to  unclear  or  poorly  structured 
Dynamic  Help  messages  where  users  cannot  find  the 
information  they  seek  because  it  is  ambiguous  or  is  hidden 
in  the  body  of  the  message. 

The  ACIFS  program  has  many  user  interface  deficiencies, 
but  it  does  allow  the  user  to  use  trial  and  error  in  task 
performance.  Many  older  computer  programs  are  very 
unforgiving  of  user  errors  or  guesses  and  would  interrupt 
task  sessions  with  the  introduction  of  unacceptable  data 
entries.  In  ACIFS,  trial  and  error  is  the  main  competitor 
against  Dynamic  Help.  Therefore,  the  author  concludes  that 
Dynamic  Help  would  be  especially  useful  in  older, 
unforgiving  programs  where  trial  and  error  is  not  permitted 
and  cannot  compete  with  Dynamic  Help. 

Although  data  on  Dynamic  Help  use  at  the  menu  selection 
screens  was  not  collected,  the  author  concludes  (based  on 
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personal  observation)  that  Dynamic  Help  did  not  appear  to  be 
helpful  overall  for  menu  navigation.  Users  clearly  had 
difficulties  in  menu  selection,  regardless  of  whether  or  not 
Dynamic  Help  was  available,  especially  in  deciding  the 
correct  submenu  item  with  which  to  begin  a  task.  This  was 
especially  true  for  the  WANG  users  in  certain  cases  where 
the  ACIFS  program  differed  from  the  WANG  program.  However, 
users  preferred  trial  and  error  over  Dynamic  Help  because  it 
seemed  easier  and  quicker  to  explore  all  promising  menu 
choices,  recognizing  the  desired  data-entry  screen  by 
viewing  candidate  screens,  rather  than  accessing  and  reading 
Dynamic  Help  first.  Difficulties  in  menu  selection  could 
have  been  prevented  by  a  rewording  of  the  choices  listed  on 
the  menu,  or  by  clearer  descriptions  of  menu  functions  in 
the  Dynamic  Help  messages. 

Dynamic  Help  is  not  useful  for  situations  where  the 
user  does  not  realize  that  a  challenge  to  efficiency  or 
accuracy  is  present.  If  a  user  does  not  know  that  Dynamic 
Help  can  assist  him  at  a  particular  point,  then  Dynamic  Help 
will  not  be  accessed  and  therefore  is  not  useful.  Dynamic 
Help  cannot  increase  efficiency  and  accuracy  unless  the  user 
recognizes  that  the  Dynamic  Help  message  contains  helpful 
information. 

5. 2. 2. 5  Diagnostic  Conclusions  Based  on  User  Opinions. 
Based  on  the  results  of  Question  #2  of  the  Post-Test 
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Questionnaire,  the  author  concludes  that  users  seek  help  for 
Procedures  more  often  than  Understanding  the  Meanings,  which 
is  sought  more  often  than  General  Orientation.  Based  on  the 
observation  that  users  spend  too  much  time  reading  help 
message  text  trying  to  find  the  information  they  seek,  the 
author  feels  that  a  restructuring  of  the  menu  messages  may 
assist  the  users  in  locating  the  information  they  seek  the 
most  often.  In  particular,  the  results  indicate  that  users 
would  prefer  to  see  the  local  portion  of  the  Dynamic  Help 
message  before  the  global  portion. 

The  results  from  Post-Test  Questionnaire  Question  #1 
indicate  that  users  sought  help  most  for  the  What  to  Enter 
category  which  is  normally  listed  near  the  end  of  the  data- 
entry  Dynamic  Help  messages,  causing  users  to  waste  time 
searching  through  most  if  not  all  of  the  help  text  for  the 
information  they  sought.  The  author  concludes  that  the 
Dynamic  Help  messages  could  be  improved  by  a  restructuring 
of  the  data-entry  Dynamic  Help  messages,  both  by  a  reversing 
of  order  in  the  sequence  of  the  global  and  local  portions 
and  by  resequencing  the  local  portion  to  present  the  most 
sought  after  information  first.  Also,  since  users  clearly 
sought  Dynamic  Help  for  What  to  Enter  information,  which 
usually  contains  lists  of  items  from  which  the  user  must 
choose,  the  author  concludes  that  users  might  be  aided  even 
more  by  having  a  separate  help  key  for  only  the  list  of 
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choices.  With  a  separate  key,  users  who  desire  a  list  may 
gain  quick  access  to  the  list  without  being  required  to  read 
a  complete  nelp  message. 

5.3  Recommendations 

5.3.1  Improvements  to  the  Current  Implementation 

Based  on  the  results  and  conclusions,  the  author 
recommends  the  following  improvements  to  the  current 
implementation  of  Dynamic  Help  in  ACIFS. 

To  reduce  the  overall  time  users  spend  using  Dynamic 
Help  in  menu  selection  screens,  the  author  recommends  that 
Dynamic  Help  messages  be  restructured  to  appear  in  the 
following  sequence: 

I»ocal 

Help  with  Procedures 
Meaning  of  Screen  or  Function 
Global 

General  Orientation 

Also,  functional  descriptions  of  the  menu  items  should  be 
clarified  or  reworded. 

To  reduce  the  overall  time  users  spend  using  Dynamic 
Help  in  data-entry  screens  and  to  infoirm  users  when 
decision-making  information  is  available,  the  author 
recommends  that  data-entry  Dynamic  Help  messages  be 
restihictured ,  that  a  separate  "List  of  Choices"  key  be 
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provided  to  users,  and  that  a  signal  or  "flag"  be  created  to 
appear  on  the  screen  whenever  the  List  of  Choices  is 
available  for  a  field  the  user  is  currently  in.  This  will 
also  increase  help  use  on  occasions  the  user  may  otherwise 
not  recognize  a  need  for  help.  For  example,  in  the  Special 
Issue  field  in  Task  11,  the  user  must  answer  the  query  with 
*'Y"  or  "N" .  If  the  "flag"  appeared  upon  entry  to  that 
field,  the  user  would  be  reminded  that  a  List  of  Choices  is 
available  to  assist  him  in  his  decision.  Without  the 
appearance  of  this  "flag",  the  user  does  not  realize  that 
decision-making  assistance  is  available. 

As  recommended  above,  the  data-entry  Dynamic  Help 
messages  should  be  restructured  as  follows: 

JUpcal 

What  to  Enter 
How  to  Enter 
Location 

Movement  within  Screen 
Global 

How  to  Enter 
Location  and  Function 
Movement  between  Screens 
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All  of  the  What  to  Enter  portions  of  the  Size  Dynamic 
Help  messages  should  clearly  indicate  how  a  user  is  to 
decide  which  datum  to  choose.  For  example,  in  the  LIN 
C07440  Size  field,  the  listing  of  acceptable  size  entries 
should  also  include  an  example  of  translation,  such  as  5-1/2 
Regular  =  055R.  It  might  also  be  helpful  for  this 
translation  information  to  be  present  in  the  List  of  Choices 
message  for  the  Size  fields. 

Since  the  author  has  recommended  that  global  portions 
of  the  Dynamic  Help  messages  should  follow  instead  of 
precede  the  local  portions,  it  may  be  helpful  to  highlight 
(in  color)  the  portions  of  the  global  message  that  include 
general  instructions  on  how  to  fill  out  a  worksheet  in  the 
data-entry  Dynamic  Help  messages.  This  would  be  especially 
helpful  for  users  who  do  not  understand  how  to  move  the 
cursor  to  another  portion  of  the  worksheet  and  need 
information  on  use  of  the  ESC  key  or  other  special  keys. 

If  practical,  all  Dynamic  Help  messages  should  be 
reviewed  to  ensure  clarity  and  completeness  of  confusion- 
reducing  information.  It  may  also  be  helpful  to  use  color 
to  highlight  those  key  points  of  information  in  Dynamic  Help 
messages  for  those  parts  of  the  ACIFS  program  where 
interface  deficiencies  are  the  greatest.  For  example,  for 
the  Task  15  Subtask  2  DOQTY  challenge,  users  were  very 
confused  about  what  quantities  to  enter  in  the  DOQTY  column 
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for  those  items  not  currently  being  issued.  It  would  be 
very  helpful  if  the  help  sentence  for  that  particular 
confusion  was  highlighted  so  that  the  user  would  locate  the 
infoirmation  quickly. 

5.3.2  Future  Research 

In  future  user  testing  of  Dynamic  Help  applications, 
the  user  tests  should  be  carefully  controlled  so  that  user 
accuracy  and  performance  times  more  closely  resemble  those 
of  an  actual  job  setting. 

This  thesis  makes  conclusions  about  the  effectiveness 
of  Dynamic  Help  as  defined  by  usahility  in  a  pre-existing 
computer  program.  The  next  phase  of  research  would  be  to 
focus  on  the  efficiency  of  providing  Dynamic  Help.  It  might 
be  useful  to  test  the  efficiency  and  the  effectiveness  of 
Dynamic  Help  messages  that  were  automatically  specified  from 
requirements  documentation. 
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APPENDIX  1 

AUTOMATED  CENTRAL  ISSUE  FACILITY  SYSTEM  (ACIPS)  FUNCTIONS 
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AUTOMATED  CENTRAL  ISSUE  FACILITi'  SYSTEM  (ACIFS) 


GIF  Main  Menu 

■  1  •  Ma*l«r  FUc  MaloUaaoce  M«ou 

2  •  A4i>*sUDcol  TraUMCltoos 

3  •  lovcoiory  Meow 

4  •  A»  R*<|u<f*d  Reports  Meeu 
5-  iuiderOojrProccssUig 

I  6  •  Documeot  KeRivUr  Aciioas  Meow 

II  •  Clothioi;  Turn  lo  Process 
8  -  Oothioi*  Issues  Process 
')  •  Tenp  Looo/Hsod  keccipi  Process 
X  •  Esll 


Master  File  Maintenance  Menu 

1  •  Stuck  File  MamtcaoDcc 
M  •  A<kl  Slock  Record 

1.2  •  Update  Slock  Record 

13  -  Oeieie  Stock  Record 

14  •  Query  Stock  Records 

1.5  •  Update  Laundiv  arid  Repair 

1.6  •  Change  FSC  on  Stock  Record 
2-  Menu  File  Mutoieoaocr 

2.1  -  Add  a  Complete  Menu 

2.2  •  Add  a  LIN  to  a  Menu 
13  -  Dcleie  a  Complete  Menu 
2.4  •  Dekte  •  UN  from  a  Menu 
2Ji  •  Clunge  a  Menu 

3  -  Special  Mcau  Flic  Maloleoaoce 

3.1  •  Add  a  Complete  Speetal  Menu 
12  •  Adda  LIN  to  a  Speoai  Menu 
31  •  Delete  a  Complete  Special  Menu 
34  •  Delete  a  UN  from  a  Special  Menu 
31  •  Change  a  Special  Menu 

4  •  Lie  File  Mntaieonoec 

•1.1  •  Add  UtC  Record 

1.2  •  Update  L'lC  Record 

4.3  •  Dekie  UlC  Record 

5  -  Ckthiop  Flic  Maiatcanoee 

5.1  -  Update  individual  Oothmt  Profile 

5.2  ’  Query  Individual  Coihing  Record 

6  '  OUlmUoo  Baloeccs  File  Muioieouoec 

6.1  ••  Balance  for  Fdcdl  Year 

6.2  -  Monthly  Balances 

6.3  •  Addifional  funds 
.X  •  Eail 


Adjustment  Transactions 

1  •  Stptemal  •!  ChufK** 

2  •  ReperipfStrrvcy 

3  •  Call  CeMaeiian 

4  •  LaiermI  Tnoater  Out 
3  -  l.jitcf«l  TrMttkr  to 

it  •  AdCBSOtslvMivc  AdjuatOMOta 
61  •  Adyusi  Item 
6J «  Change  Stock  Number 
6J  •  Change  Unti  of  Uawe 
6.4  •  Chatige  Accounting 

6.4.1  •  Accountable  to  Non-accountabic 
64.3  •  Notv^ccouniabk  to  Accountable 
6J '  Aaicmbly 
6.6  >  OiMsicrMly 
67  •  COMtiflicd 
7  •  FuuodMilaMolluituii 
•  •  Turn  lot  lb  S&VORMO 
8.l*ToSSA 
63 '  To  ORMO 
9  .  Cooccl  .VC  R/S.  C/C 
X  'bit 


Inventory  Menu 

I  •  Priot  lovtotufy  Cpuo*  Shetu 
2 '  Fuel  lovtMbfT  Cbuoi  Tptub 
3  •  FriPl  loiliul  lovtotbry  AdjuaimecU  Hepnrt 
X-EbM 


i'  ClF  AG  FunCTion 


Aaministrative  Menu 

1  .Eolry  to  CIP  .Mam 

2  •  lostollotloo  UAkiu*  OaU 

2.1  •  Add  tnsutuion  Unique  Data 

2.2  -  Change  Installation  Unique  Data 

3  •  ClF  File  Comellofis 

31-  Print  Document  Kegisur  Blank  or  Nepirve  hietd  Report 

3.2  •  Correct  Blank  or  Negative  Fields  in  DtXMmeni  Register 
3  3  •  Correct  Due  In  Quanoivon  Propenv  Hoo« 

3.4  .  Correct  Blank  Ouamitv  Possessed  t>y  Sr>ldiers  on  Ptopeny  Book 

3.5  -  Ocktc  Invalid  Clothing  Item  Records 

3.6  -  Correct  Due  Out  Quantify  on  Property  Book 

3.7  -  Correct  Temporary  Uoan/Hand  Hecctpi  Quantity  on  Properry  Book 

3.8  -  Set  tor  Temporary  Loun/Hand  Keceipi  Quant  ity  on  Propery  Book 

4  .  iDlUal  Start  Up  nf  ClF  Fuociteoa 

4.1  •  Create  Aukomated  Oolhing  RccoiOs 

4.2  •  Stock  Fik  Mainunancc 

4.3. 1  •  Add  Stock  Recotd 

4.2.2  -  Update  Stock  Record 
4.2J  -  Delete  Stock  Recotd 
4.24  .  Query  Stock  Record 

4.2.5  •  Update  Laundry  and  Repair 
4.2.6-  Change  FSC  on  Stock  Rcforri 

4.3  -  UTC  File  Maintenance 

4.21  -  Add  UIC  kccord 
4J.2  -  Update  UIC  Record 
4J.3  -  Dekic  UiC  Record 

4.4  -  Create  iniiial  Document  Regiiier  Transactions 

5  •  Frlfil  Flfsal  loveDlary  AdyustiiMOl  Hcpuri 

6  •  purge  Funcliaa 

6.)  •  Putge  Completed  Document  Register  Records 

62  -  Purge  Summary  of  Operations  Records 

63  •  Purge  Obligaiion  and  Baunce  Records 

7  •  TraaaferFuaciHNU 

7.1  •  Load  UNlTTranafcTong  into  ClF 
7.3  -  Load  INT)IV|DUAL  Transferring  into  OF 
X-Ealt 


As  Required  Repons  Menu 

1  •  Stock  Keporu 

l.j  •  Stock  PropefTv  Book 

1.2  •  Contingency  items 

1.3  •  Siockaee  Reorder 

I  4  .  Quantity  Pouesacd  By  SoWiers 
1 J  •  Stock  items  for  issue 

1.6  •  Laundry  and  Repair 

17  •  Selected  Stock  .Master  Record 

1.8  •  Dekied  Stock  Records 

1.9  •  Stock  On'Hand  Totals 

2  •  DocuomoI  Rcgisicr  Repucu 

2.1  •  Open  Documents  bv  Document  Numocr 

12  •  OC  R/S.  S/C  Adfusiment  Totals 

13  •  Items  Found  on  installation 

14  •  Due  In  Document  Rcgisicr  by  NSN 
2J  •  Document  Register  bvSekcied  Dates 

2.6  •  Summary  of  Issues.  Tumm  and  DXs 

3  •  Lail  Repuru 

3.1  -  Unit  Reconciliaiton 

3.2  •  Quaneriv  Reconciliation 

3.3  •  Temporary  Loans/Hand  Receipts  by  UIC 

3  4  .  Cunsoltdated  Due  Out  Items 
3J '  Unit  Transfer 

4  •  Cktbrng  Repofu 

4.1  -  Oothmg  ProTiks 

4.2 '  Individuals  Due  to  DER05 

4  3  •  Individuals  Past  Due  OEROS 

4  4  -  Indwidual  Transfer 
5 '  Other  KepufU 

5.1  •  Attfhonted  Clothins  Menift 

5.3  •  Monthhr  Obligation  and  Ceiling  Balance 

5.3  •  Yearly  Oblisanon  and  Ceiling  Balance 

5  4  .  Summary  oi  Operations 

1 X .  E»ii  '  _ 
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APPENDIX  2 

ARCHITECTURE  AND  FORMAT  OF  DYNAMIC  HELP  MESSAGES 
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APPENDIX  2;  Excerpt  from  Young,  1990b. 


Specifications  for  other  conversions  bevond  CIF  will  follow  late: 

2.  Generic  Requirements 

Dynamic  Help  messages  are  built  rather  than  retrieved.  .A.';  is  true  of  all 
interactive  systems,  the  interaction  cycles  among  three  superstates,  idling,  inputting  and 
processing  (see  Appendix  2).  Dynamic  Help  is  to  be  available  ouring  the  idling  and 
inputting  superstates. 

To  work  most  effectively,  a  Dynamic  Help  system  requires  the  basic  interaction 
design  to  allow  two-step  (selection  and  closure)  rather  than  onJv  one-step  (selection  = 
closure)  user  actions.  A  one-step  or  premature-closure  menu  selection  action  e.’cists.  for 
example,  where  pressing  a  single  key  both  selects  the  menu  item  and  invokes  it.  If  the 
user  can  first  highlight  an  item  (say  by  using  arrow  keys  or  a  mouse  )  and  then  invoke  the 
highlighted  item  (say  by  <CR>),  Dynamic  Help  can  be  specifically  available  for  each 
menu  item;  otherwise  one  message  must  cover  all  items.  A  one-step  or  premature- 
closure  data-entry  action  exists,  for  example,  where  filling  the  field  automatically  causes 
closure.  A  two-step  entry  (typing,  then  <CR>)  prevents  accidentally  getting  beyond  the 
point  where  Help  is  needed. 

Effective  Dynamic  Help  also  requires  a  true  neutral  state  to  be  provided  on  all 
screens^a  state  in  which  no  object  is  highlighted.  This  does  not  mean  there  should  not 
be  a  default  highlighted  object  (as  having  the  cursor  on  the  assumed  first  data-entry  field 
upon  screen  entry),  but  only  that  it  should  be  possible  to  get  to  a  neutral  state, 

2.1  Global,  Context  and  Navigation  Sentences 

During  idling,  when  the  interaction  is  in  a  neutral  superstate  waiting  for  user 
input,  there  is  a  comext,  which  for  the  user  is  charaaerized  by  a  sense  of  "where"  the 
user  is;  what  screen  is  displayed,  the  history  of  recent  navigations  among  contexts, 
whether  any  data  has  been  changes  "here"  yet,  and  the  identity  of  "loaded"  c. 
immediately-available  data. 

For  a  screen  in  which  there  is  no  true  neutral  state,  the  first  sentence  in  the 
Dynamic  Help  message  is  a  global  sentence,  which  is  a  Ready  to  sentence  exactly  like  the 
context  sentence  described  in  the  next  paragraph,  except  that  it  refers  to  the  neutral 
context  rather  than  the  comext  of  the  highlighted  item. 

For  each  neutral  superstate,  a  Dynamic  Help  sentence  staning  with  the  phrase 
Ready  to  will  be  followed  by  one  or  more  context-dependent  imperative-verb  phrases,  the 
first  one  of  which  will  describe  the  most  typical  user  action,  such  as  "issue  clothing  to  a 
<G>  soldier,"  where  <G>  will  be  a  distinguishing  phrase  if  there  are  separate  screens 
for  issuing  clothing  to  separate  kinds  of  soldier  or  under  separate  circumstances.  If  the 
same  context  supports  other  typical  aaions  (except  the  standard  ones  listed  below),  one 
or  more  or  phrases  will  follow,  such  as  "or  revise  a  <G>  soldier’s  <H>  size"  (where 
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<H>  names  the  class  of  clothing). 


Note  that  although  the  purpose  of  the  Ready  to  sentence  in  io  oncni  ihe  uNei  ic 
"where"  the  user  is.  the  sentence  says  what  the  user  can  do.  .NaniCN  lor  coniexi  piacc' 
such  as  "Clothing  Issue  Form"  are  not  as  useful  or  transparent  as  phrases  tnai  descrine 
the  functions  they  support,  and  the  names  are  often  already  on  displas. 

After  the  Ready  to  context  sentence  will  appear  a  navigation  sentence  has  ing  tuo 
clauses.  The  back-navigation  clause  is  ‘To  return  to  <  1  >  <J>'  where  <  1  >  is  the 
imperative-verb  phrase  from  the  last  previous  context,  such  as  "choose  soldier  in¬ 
processing  function."  and  <J>  is  a  set  of  instruction  phra.ses  that  give  the  method  of 
losing  or  reversing  all  data  changes  and  the  method  of  keeping  all  data  changes 
(suppressed  when  no  data  has  been  changed  since  context  entry  ).  The  ioniard-navigaunr 
clause  IS  'To  go  to  <K>  <J>"  where  <K>  is  the  imperative-verb  pnrase  lor  the 
typically-next  context. 

2.2  Meaning,  Domain,  Content  and  Format  Sentences 

During  inputting,  when  the  system  is  in  an  input-procedure  superstate,  the 
following  sentences  arc  optionally  provided: 

Meaning  Sentence.  This  tells  the  semantic  meaning,  in  application-environmem 
language,  of  the  datum  currently  being  entered.  As  an  example,  in  correcting  a  repori- 
of-survey"  in  the  CIF  system,  the  TOTAL  datum  could  be  interpreted  by  the  user  as 
meaning  the  correaed  report-of-survey  total  (wrong)  or  as  the  amount  to  be  subtracted 
from  the  report-of-survey  and  thus  added  to  inventory  (correct). 

Domain  Sentence.  This  gives  limits  on  the  datum  being  entered,  such  as  whether 
it  is  textual  or  numeric,  whether  it  has  upper  or  lower  limits  or  impermissible  values,  and 
the  maximum  or  required  number  of  charaaers. 

Content  Sentence.  This  gives  a  list  of  legal  or  illegal  entries.  (Because 
recognition  is  easier  than  recall,  good  interface  design  provides  that  a  user  should  be 
able  to  view  a  list  when  asked  to  make  an  input  that  essentially  chooses  from  one.) 
Dynamic  Help  should  not  provide  a  list  when  the  list  is  intentionally  withheld  for  quality - 
control,  security  or  validation  purposes. 

Format  Sentence.  This  exempliOes  or  describes  the  acceptable  format  of  an  entry  . 

23  Procedure  Sentences:  Complete,  Pre-Completion  Correction,  Undo,  Key 
Effect 


Completion  Sentence.  During  data  entry,  this  sentence  describes  how  the  entry 
can  be  completed  or  aborted.  If  there  are  quirks  such  as  unexpected  behavior  of  the 
backspace  key  or  automatic  completion  upon  filling  the  field,  this  is  the  sentence  that 
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informs  the  user  of  quirks. 


Pre-Completion  Correction  Sentence.  During  data  entr>'.  this  sentence  describes 
how  the  entry  can  be  changed  before  being  completed  (such  as  by  using  the  backspace 
key),  if  the  description  is  not  included  in  the  completion  sentence. 

Undo  Sentence.  During  data  entry,  this  sentence  describes  how  the  entn,.  i; 
completed,  can  later  be  undone  (reversed). 

Key  Effect.  During  data  entry,  this  sentence  describes  what  would  be  the  effect  of 
pressing  standard  keys  such  as  ESC,  DEL,  BREAK,  FI.  or  any  other  keys  that  have  a 
globally  consistent  meaning  that  may  be  misinterpreted  or  inconsistently  implemented 
with  respect  to  this  data  entry. 

3.  General  GIF  User  Requirements 

CIF  is  DBMS-based  and  is  organized  by  function,  where  almost  all  of  the 
functions  are  to  add,  update,  delete  and  query  database  data.  CIF  functions  are 
organized  by  hierarchial  menus,  with  a  single-page  menu  screen  for  the  main  menu,  a 
single-page  menu  screen  for  each  submenu,  and  a  multi-page  forms  screen  for  each  leaf  of 
the  menu.  Thus  almost  all  the  screens  are  of  two  types-menu  and  form.  The  menu 
screens  were  developed  using  the  INFORMIX  menu  utility,  and  the  forms  screens  were 
developed  using  the  INFORMIX  forms  utility. 


3.1  Existing  Help  and  Error  Message  Files 

CIF  has  an  existing  system  of  static  Help  message  files.  The  Dynamic  Help 
system  will  neither  replace  the  existing  Help  message  files  nor  supplement  them.  The 
existing  Help  files  and  access  path  should  be  left  in  place,  but  the  new  Dynamic  Help 
system  will  not  assume  use  of  the  existing  Help  system,  and  may  write  over  some  of  the 
existing  system’s  messages. 

32  Dynamic  Help  for  GIF  Menu  Screens 

Menu  selection  protocol  is  normally  one-step  in  CIF;  the  user  does  not  first 
highlight  a  menu  item  and  then  select  it  However,  two-step  selection  is  also  provided 
using  arrow  keys  and  <CR>.  Dynamic  Help  for  menu  selections  will  be  available  only 
for  two-step  seleaion. 

The  Ready  to  sentence  for  menu  screens  will  say  "Ready  to  selea  <L>’'  where 
<L>  is  the  designator  of  the  highlighted  menu  item  (these  designators  are  numbered 
1.2,.,„X,  X  being  the  exit  item),  followed  by  the  phrase  "to",  followed  by  the  imperative- 
verb  phrase  for  the  selected  screen’s  context  sentence  (<  K  > ).  Example:  "Ready  to 
select  1  to  post  a  Statement  of  Charges."  The  navigation  sentence  is  "To  return  to  <  I  > 
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wh  .  1  -jc  imperaiive-verb  pnrase  from  the  last  previous  screen's  context 

sentence,  followed  by  the  instruction  phrase  "move  to  X  and  press  RETURN  "  (X  is 
always  the  back-navigation  menu  item),  followed  by  ",  To  make  a  different  selection, 
move  to  it." 

3.3  Dynamic  Help  for  CIF  Form  Screens 

CIF  form  screens  are  for  data  entry  and  provide  a  window  currently  being  used 
for  informative  messages  positioned  beside  their  respective  data  fields.  Dynamic  Help 
messages  will  replace  these  messages  (write  over  them)  and  will  normally  use  the  whole 
window  from  the  top  instead  of  just  the  area  to  the  right  of  the  highlighted  item. 

The  Ready  to  sentence  for  forms  screens  will  say  "Ready  to  provide  <M>  "  where 
<M>  is  a  supply-language  descriptor  of  the  datum.  The  navigation  sentence  will  be  ad- 
hoc,  telling  how  best  to  get  to  the  last-previous  and  next  data-entry  items:  for  example, 
on  the  Complete  Initial  Issue  screen  there  must  be  a  phrase  or  clause  telling  how  to  get 
to  the  next  size  datum  entry  field,  since  arrow  keys  go  only  to  the  next  line  whereas 
entries  occur  only  on  a  few  lines.  The  back-navigation  clause  will  explain  the  sometime.s 
complex  rules  of  what  data  is  saved  and  what  is  lost  when  the  DEL  or  ESC  keys  are 
pressed. 

The  meaning  sentence  will  be  provided  for  each  entry  by  a  logistics  officer  (CPT 
Wolven). 

The  domain  sentence  will  start  "Entry  must  be"  and  will  continue  with  a  list  of 
descriptors  such  as  "positive"  (mention  the  word  "negative"  whenever  negative  entries  are 
legal).  In  CIF,  the  lengths  of  fields  are  already  on  display,  so  the  maximum  or  required 
number  of  characters  in  the  entry  need  not  be  mentioned. 

The  content  sentence  will  give  a  list  of  legal  or  illegal  entries  (obtained,  where 
appropriate,  via  an  SQL  query)  or  directions  on  how  to  determine  legal  or  illegal  entries. 
CIF  withholds  some  data  intentionally:  social  security  numbers  of  soldiers  are  not 
displayed,  and  in  posting  inventory  counts,  the  on-hand  data  with  which  the  counts  are 
compared  is  not  displayed.  We  will  not  second-guess  the  CIF  designers  in  cases  such  as 
this;  where  omission  is  intentional,  we  will  not  supply  lists. 

The  format  sentence  will  give  an  example  of  the  correct  format  or  describe  a 
correct  format,  whichever  is  more  informative.  Not  all  legal  format  variations  need  be 
covered,  but  the  example(s)  should  show  the  most  efficient  (such  as  omitting  leading 
zeros).  Where  CIF  entries  have  the  quirk  of  being  automatically  completed  upon  filling 
of  the  field,  this  should  be  noted  (since  <CR>  can  unintentionally  skip  the  next  field). 

The  procedure  sentences  will  be  ad-hoc.  to  tell  how  to  abort  the  entry',  how  to 
complete  the  entry,  how  to  make  a  pre-completion  correction  (CIF  has  quirk7  backspace 
rules),  bow  to  undo  or  reverse  the  entry  without  affecting  other  data,  and  what  the  effect 
would  be  if  the  user  pressed  the  DEL,  ESC.  <CR>  or  other  globally-meanmgful  keys 
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now. 


If  it  proves  possible  to  invoke  dynamic  Help  while  in  a  field  after  individual 
keystrokes  have  been  made,  these  specifications  will  be  revised  to  e.vpioit  that  capabilii\. 

4.  Dynamic  Help  System  Standards 

4.1  Context  Monitoring 

DBMS-based  interactive  programs  maintain  an  identifier  for  the  current  screen 
and  for  the  last  previous  screen.  The  Dynamic  Help  system  must  be  able  to  query  these 
identifiers.  The  Dynamic  Help  system  will  also  keep  in  its  own  knowledge  base  (which 
should  be  ordinary  tables  in  the  database,  not  programmer-defined  ad-hoc  constructs)  a 
table  of  ordinarily-next  screens,  the  data  for  which  w-ill  be  provided  by  a  logistics  officer 
(CPT  Wolven)  for  the  CIF  program.  For  the  CIF  program,  these  three  identifiers  and 
the  identity  of  the  currently  highlighted  screen  object  (the  highlighted  menu  item  for  a 
menu  screen,  or  the  cursor-located  datum  for  a  forms  screen)  constitute  the  entire 
context.  (In  other  programs,  the  recent  user  actions,  session  history,  and  identity  of 
loaded  data  files  could  also  be  part  of  the  context. 

The  CIF  Dynamic  Help  system  will  have  continual  access  to  the  current-screen 
identifier,  the  previous-screen  identifier,  the  ordinarily-next-screen  identifiers,  and  the 
currently-highlighted  SCTcen  object. 

4.2  State  Monitoring 

For  CIF  and  any  other  DBMS-based  software  systems,  the  data  state  of  the 
system  will  always  be  available  to  the  Dynamic  Help  system  through  its  capability  to 
issue  SQL  queries  to  the  database. 

43  Message  Architecture,  Generation  and  Assembly 

Dynamic  Help  messages  are  built  of  sentences  that  contain  clauses  that  contain 
phrases  (informally  we  use  the  word  "phrase"  sometimes  to  mean  a  set  of  phrases).  Each 
sentence  has  a  generic  name;  these  are 

Context  (Ready  to) 

Navigation  with  back-navigation  and  forward-navigation  clauses 
Meaning  (semantic) 

Domain 

Context 

Format 

Completion 

Pre-Completion  Correction 
Undo 
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k.  ,  ^ifect 

Each  phrase  is  either  a  constant  phrase  (such  as  Ready  to)  or  a  variable  phrase.  Phrases, 
clauses  and  sentences  are  allowed  to  be  null. 

A  whole  Dynamic  Help  message  consists  of  all  ns  defined  sentences,  in  fixed 
order.  The  variable  phrases  are  materialized  into  text  by  query  and  assembled  with  the 
constant  phrases  to  form  the  single  text  object  that  is  the  message. 

4.4  Message  Display 

In  general,  it  is  best  for  Dynamic  Help  messages  to  use  a  pop-up  utility  bv  which 
they  can  temporarily  overwrite  arbitrary'  di.splays. 

For  CIF.  it  is  believed  there  is  no  pop-up  capabilirv.  but  a  message  window  is 
provided.  Dynamic  Help  messages  will  be  wTitten  in  this  window  when  invoked,  and 
removed  upon  the  next  user  action. 

When  necessary,  existing  messages  should  be  suppressed  if  they  interfere  or  if 
Dynamic  Help  messages  interfere  with  them.  Most  will  be  made  redundant  by  Dynamic 
Help.  In  CIF,  there  are  many  that  persist  too  long;  rather  than  providing  for  timely 
removal,  simply  suppress  them. 

Suppression  of  interfering  or  competing  messages  should  be  the  only  Dynamic 
Help  programming  required  outside  the  EKmamic  Help  system  itself. 

4.5  Invocation  of  Dynamic  Help 

A  standard  invocation  for  Dynamic  Help  will  be  the  FI  key.  This  should  be  used 
for  CIF  if  possible,  but  not  if  it  requires  detailed  programming  to  redefine  other  existing 
uses  of  the  Fl  key. 

CIF  uses  the  FIO  key  for  static  Help,  which  does  not  compete  or  interfere  with 
Dynamic  Help  and  should  be  left  as  is. 

Upon  invocation,  the  Dynamic  Help  system  generates,  assembles  and  displays  the 
message.  Upon  the  next  user  action  (next  keystroke  preferably,  or  next  completed  entry) 
the  message  is  removed. 

4.6  Tone 

Where  details  or  content  of  messages  is  left  to  the  programmer,  this  general 
guidance  applies:  the  proper  tone  is  empty  of  any  meta-conununication  and  direaly  tells 
the  user  what  to  do.  Vocabulary  should  be  that  of  the  application;  CIF  concerns  shoes 
and  inventory,  not  windows  and  menu  items. 
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APPENDIX  3 

SAMPLE  ACIFS  DYNAMIC  HELP  MESSAGE  SPECIFICATIONS 
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.pCATION  information; 

A.  Current_Screen:  (LV\^^r\C^  Notni?g.r _ 

.  E-i  TrevioLis  Screen:  Admirns-hru-Hy^e.  /\djUS-fmgrTt-  lVk.nL< 

C  iJrd  Ne::  t  icreeu  y  _ _ _ 


t^EADIN6:  _ 


SLOBAL  CONTEXT:  liilierr*  the  i  ser  <■  and  wnat  hi?  -.mer  >  an  do, 

"Readv  i:o  <CH>4ki,  of  gn  <i-rM>  on4hf  <?&'>.  <aoTnt' 

nU  <-k'>  USKJ.  <  £.30  -to  <^PgOC><  RCDS  >  .  <0£L'>  . 

BACrUJAftD  IJAVIuATTGII:  'Tu  r  eturn  'c  ^  '> _ 

FOrvWMRD  HAVI!5ATIUN:  '  Tu  no  -□  C  .  ^  ^  2 _ • 

COMPi-ETIOl-):  'T<-i  i.omoifft'?  «n  i»nt. r-  .  <^CR'> _ 


QUIRKS;  "NllTE:  <  Fa.TaI,'? 


r  FF. -CDnF'LETK.ltl  S'uF.'REC  I  lOM;  "Ti:«  iMdk'?  i  ■.  arrnrtiDn  kIuii.’  >*♦* 
I :  -  til  I  in  The  ,>nrr'-  .-lolu.  <BAJZK> _ 


UNDl):  "Tu  cn«r  e  a  :orrncTi  on.  fJU  ^ 


Er  EFFECT: 
i:E"T 
r- 3C 
•.•EL 

14 


IFFTC': 

<NU? 
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GQPT  AVAILABLE  TO  DTIC  DOES  EOT  FEEMIT  FULLT  LSQIBLE  EEFRQDUCTIOS, 


1  _ HELP  Message  Soeei -f  i  car  i  ons  ( I NFUT  -  LOCAL)  Page _ 

LOCATION  INFORHAT I ON : 

<A>  Curpjent  ^c^een  :  ^~r 3.-re TT'en i  o-  TO^S _ 


<X>  Current  Data  Object;  AJ5AJ 

Last  Data  Qbiect:  <'^ty  > 

l'je::c  Data  Oo  lect  Oi/pD-L.t'U 

HEADINS:  X  > 

LOCAL  CONTEXT^: 

"Reitclv  to  ^  EkTTEIS  >  4v>  ly^  Qy-  ^  ITN-  >  I'.RTg!^  DT  — r-L  > . 


Ni^  I  GAf  ION 

: 

DACKWARD: 

"To  return 

to  D,.:.  ^UU> 

Z-iACI\WARD: 

"To  return 

to  V  . 

FORWARD : 

"To  ao  to 

CNU> 

FORWARD: 

"To  ao  to 

<£W7a®>  -ex'?  <^> 

MEANT No;  ' 

'<m  > 

DOMAIN: : i ■ 

necessarv. 

■  <-ACC£P7  >  ^2£>  C  i  >  <P3£  > 

CONTENT :  '.List  or  Dire 

cti ans J 

_I5T  DIRECTIONS 

lUuerv  04*1  i  .  _ 

or 

<tJU>  - 


FORMA  I:  ..E::«imDie  ur  L-escri  oti  Hn> 

<eXAM'p  or  I 

[S:*ii<5oii'?‘yi2?-^J  ZHUZZI 
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C0F7  AVAILABI£  TO  DTIC  DOES  NOT  PEBASIT  FULLY"  LEGIBLE  aEPRODUGTIOl^ 


'f 


1-HiJCE.DlJKE  -3EI  iTEHCEE  : 

CDMHLEI  1  OivJ :  '  1  c  ’  itiisri  t'Pr  r-r  i  n::  ^ 

'"111.  •rfr-.  .  _  f  4 'J  y 


OUIEt  E;  vi^cHC)  aaauc’  ■  e'.  K'.r  .;on;oi  7C’.a'  .iucp  •  i  .  :  ic  'i-  ■  ;vJ 
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00P7  AVAILABLE  TO  OTIC  DOES  HOT  FEBMIT  FULLV  LEGIBLE  BEFRODUCnOSj 


1 


Diction«ry  0+  HELP  Hess«ae  Cooes 


CODE  DEFINITION /MEANING 


•'  '  “  Phrase  C  I 


:100>  Up  to  100  NSNs  allowea 


<  S<  > 

<A> 

<  ACCEPT 
<ADDL> 

<:  APP  > 
<AR> 
<AUTHV 
< AUTO 5 
<B> 

':bacfc> 

<BCFCSP  i 

:bfr> 

<BLAK> 

<0 

:CARR> 

<CH> 

<CLO> 

<CMPD> 

iCoriJ 

<C0MP> 

<CR> 

<CUR> 

<DEL,> 

<DES> 

<DESD> 

<DN> 

<DNR> 

<D0C> 

<D0CS> 

<DR> 

<DU> 

<DX> 

<ELSE> 

< ENTER > 

<ENTRS> 

<ERR> 

<ESC> 

<EXAM> 

<EXCH> 

<FATAL> 

<F1LA> 

<FILT> 

<F1n> 

<6> 

<HLT> 

<1ND> 

<INDS> 

<1NF0> 


and 

•  Current  Screen  De-fxnicion  / 

Acceptable  entries  are 
addi  ti  onal 
1 -f  applicable 

See  AR7C5^50  -for  soeci-fjc  cooes 

author 1  ted 

automated 

♦Previous  Screen  De-fimtion 
use  BACKSPACE  and  overstnue  incorrect  oata 
press  BACKSPACE  Kev 
be+ore  completino 

BACFCSPACE  Key  does  not  erase  data 
♦Ordinarily  Ne>:t  Screen 
carried 
chanoe 
clothino 
compl eted 
complete 
complete  all 
press  Return  Ftev 
cursor  jumps  to 
press  DEL  (Delete)  Key 
desi r i na 
is  desired 

press  DOWN  Arrow  Kev 
document  number 
document 
documents 
document  reoister 
due-out 

direct  exchanoe 
Otherwise, 
enter  the 
entries 

an  error  occurs 
press  ESC  Key 
♦♦EXAMPLE : 
exchanped 

H  a  ■fatal  error  occurs,  all  work  is  lost  and 
heentered  once  problem  is  corrected. 

FiHino  a  field  sipnals  completion 
Fillinp  the  field  Sionals  completion 
to  complete 
to  go  to 
hi ghl i ght 
individual 
i ndi Vi  dual ’ s 
i nf ormati on 


-  E>:  amp  1  e  , 
List  or 

P  or mat 
-  Query 


must  be 
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;  I N I T  '  r.  j  1 1  i  .  1 

.>J.  '  ..notner  install. on 

I TM  1 tem 

I TMS  1  terns 

I S5  1 SBUP 

ISSD  1 ssuen 

,LRAH  Lise  LEFT  KIGHI  Mrro^v  t  e/'.,  to  tuciMliotit 

..LT  press  LEFT  Arrow  Lev 

.MANU'  manual 

.MATCH  matcnes  disolaveo 

;MF  -  "M"  -for  mal  u  and  'T  1  o'  -femait 

-.NEC  necessarv 

NEED  as  neeaea 

.NEG  neoative  numoers 

NEX  ne;:t  oata  Tieid 

NOARR  Do  NOT  use  Arrow  t.evs  or.  tni=  screen.. 

NOTE  **NOTE: 

.NOUSE  Do  not  use  this  option  ir 

: NU'  ♦NULL  entrv 

.OH  letter  o  and  i..  are  oiTTorent 

.OMIT  To  omit  leadino  zeros,  enter  sicni-ficant  diaits 

ONHD  on  nand 

.OFT  desired  oction  numoe'- 

.OVEr.:  overstrike 

.PE  Prooertv  Book 

PERP  oer+orm  entrv  aoain 

'POS.--  positive  numbers 

<PR.  press 

PRN/  print 

<PROC^  process 

.PRCD.  processed 

<PRC6;  oroceasina 

;CTY  puantitv 

sQTYS^  puantities 

TQUIT  witnout  savina  entries 

<R>  to  return  to 

;RCD:  record 

'RCDS  records 

'REOPEN  Closed  documents  must  oe  reopened  usi np  option  B  berore 
usinp  tnis  option. 

<REOT  reouisition 

REGS'  reouisitions 

<RQD;  This  entrv  is  reoui red  to  process  the  transaction. 

<RTT  press  RIGHT  Arrow  t.ev 

.'SISS^  special  issue 

;SUP  •  supolv 

.SZ:^  size 

•SZS.  sizes 

TSZOT  sized 

.'SUCH'  such  as 

^TB'-  to  be 

^TI,  turn  in 

-TID'.  turned  in 

-TR.:  transaction 
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COPY  available  to  one  DOES  NOT  PBIUflT  FULLT  LEGIBLE  IVEPBODUCTIOW 


g' 


<  TRS  t  r  aniiac  1 1  on?. 

' TRN5F  transreraole 

UF'  orees  Ur  Arrow  Lev 

UPDT  current  record  wi 1 i  bo  orocesseo 

-WKSH-  worksheet 

<  X  ■  ♦Current  Datii  Field 

<Y>  *Frevious  Data  Field 

. YN  "Y"  or  "N" 

.  YNCF;  enter  "Y"  and  ores?.  Return  Kev.  Dtncrwise.  enter  "N" 

and  Dress  Return  Rev. 

\Z:  *Ne;rt  Data  Field 

V  ZE  t  ero 


C ADVICED  Entry  snould  be  ot  the  torm  "Att"  where 
"A"  denotes  a  letter  and 
denotes  a  number. 

CDATED  Format  is  (MM/DD/YY) 

C GRADED  Enlisted  -  EOl  -  E1<D 
O-f-ficer  -  001  -  010 
Warrant  -  WOl  -  W05 
Civilian  -  GOl  -  G15 
Korean  -  KOI  -  KlE 

CJDATD  Format  is  (YDDD)  .  5  Jan  91  =  1005. 

CLIND  Entry  should  be  o-f  the  +orm  "A#####"  where 

"A"  denotes  a  letter  and 
"it"  denotes  a  number. 

CMOSD  Enter  the  3  to  5  letter  code  corresponding  to  an 

individual's  Military  Occupational  Specialty,  such  as 
11B20  -  in+antry  <E5) 

956  -  military  police 

926  -  supply  o-f-ficer 

CSSND  123456799  for  US  soldier 

K23456789  tor  KATUSA 
KC3456789  tor  Korean  civilian 

CSTATUSD  Entry  should  be  o-f  the  torm  "A#"  or  "AA"  where 
"A"  denotes  a  letter  and 
denotes  a  number. 

CSISSUEI  select  unigue  MOSO  -from  aob02t 
where  SISSUE-"Y"; 

{SZ— f  >  select  unique  SI2  -from  aebOlt  where  L I  N»  if  )  ; 

{TRF>  select  unique  LIN.  NOMEN  -from  aebOlt 

where  LIN«  (select  unique  LIN  -from  aeb06t 
where  TRF»"V"); 
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COPY  AVAILABLE  TO  DTKJ  DOBS  NOT  PEKMIT  FULLV  LEGIBLE  REPRODUCTION 


cuic: 


•*  - 

**  - 


aeiect  UIC:  +ron’  acD04r. 

where  0  in  UIC  and  'O’  in  UIC; 

•elect  UIC  -frotti  aeD04t 

where  ASSK=(selcct  ma::(AESG)  from  aeb'i4t): 


Idcnti'fiec.  locatior;  or  state  (not  literal  text  ’ 
Items  should  be  highliqhteO  with  di’f-terent  text 


col  or 


133 


APPENDIX  4 

PERSONNEL  AND  EQUIPMENT  REQUEST 
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1 


Feb  91 


MEMORANDUM  FOR  Director,  AIRMICS,  ATTN;  ASQB-GM  (MAJ  Mizell), 
115  O'Keefe  Building,  Atlanta,  GA  30332-0800 

SUBJECT:  CBIET  Effectiveness  Testing  Personnel  and  Equipment 

Request 


1.  Effectiveness  testing  of  the  ISM  Computer  Based  Instruction 
Embedded  Training  (CBIET)  pilot  project’s  converted  CIF  software 
is  tentatively  4-25  Mar  91. 

2.  Request  that  Software  Development  Center  -  Atlanta  provide 
two  Wyse  Computer  terminals  and  other  supporting  hardware  for 
experimental  testing  during  the  period  4-25  Mar  91. 

3.  Request  that  25  individuals  (as  described  in  paragraph  f4)  be 
provided  to  undergo  experimental  testing  of  the  CIF  software 
conversion's  user-friendliness  during  the  period  7-25  Mar  91.  It 
is  anticipated  that  all  tasked  individuals  will  complete  a  pre¬ 
screen  survey  questionaire  on  7  Mar  91.  On  8  Mar  91,  each 
qualifying  test  subject  will  be  scheduled  for  two  4-hour  bloc)cs 
of  testing (8  hours  total  per  individual)  during  11-22  Mar  91. 
Testing  will  be  conducted  at  Software  Development  Center  - 
Atlanta,  Bldg  207,  Ft.  Gillem. 

4.  To  ensure  that  test  results  are  relevant  to  the  overall  ISM 
project,  individuals  who  will  participate  in  the  testing  should 
fit  the  following  profile: 

E3  -  E6,  76Y,76V,76C,  or  76P  (or  civilian  equivalent) 
who  has  low  or  moderate  computer  terminal  experience 
either  in  schooling  or  on  the  job. 

5.  Request  that  this  tasking  originate  from  AIRMICS.  It  is 
urgent  that  the  availability  of  the  25  individuals  be  confirmed 
by  22  Feb  91  and  that  the  names  and  phone  numbers  of  the 
individuals  be  available  by  26  Feb  91. 

6.  Points  of  contact  for  this  request  are  CPT  Renee  Wolven  (894- 
4824)  or  Dr.  Donovan  Young  (894-2321). 


DONOVAN  B.  YOUNG 
Associate  Professor 
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APPENDIX  5 
USER  TESTING  TASKS 
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TASK  llA 


Instructions; 


You  are  at  the  GIF  Main  Menu.  When  you  are  ready  to 
begin  this  task,  ask  the  System  Administrator  to  make  the 
Header  Call.  Once  the  Header  Call  is  complete,  turn  to  next 
page  and  begin  the  Task. 
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TASK  llA 


11.1  Situation; 

A  departing  soldier  has  completely  turned  in  her  OCIE 
as  indicated  on  her  DA  Form  3645s  (manual) .  She  will  be 
carrying  some  items  with  her  to  her  next  duty  station  and 
she  is  still  signed  for  them  on  her  DA  Form  3645s  (manual) . 

11.2  Requirements: 

Post  the  completed  turn-in  and  the  transferred  items  to 
the  soldier's  clothing  record. 


11.3  Attachments : 

DA  Form  3645 
DA  Form  3645-1 
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TASK  llA 


Instructions; 

Please  ensure  that  you  have  returned  to  the  GIF  Main 

Menu. 


Now,  ask  the  System  Administrator  to  make  the  Closure 
Call  for  this  Task. 

A  hard  copy  of  the  soldier's  turn  in  will  be  printed 
out.  Please  inform  the  System  Administrator  so  that  the 
printout  may  be  retrieved  from  the  printer. 
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TASK  IIB 


11.1  Situation; 


A  departing  soldier  has  completely  turned  in  his  OCIE 
as  indicated  on  his  DA  Form  3646s  (manual) .  He  will  be 
carrying  some  items  with  him  to  his  next  duty  station  and  he 
is  still  signed  for  them  on  his  DA  Form  3645s  (manual) . 


11.2  Recmirements; 

Post  the  completed  turn-in  and  the  transferred  items  to 
the  soldier's  clothing  record. 


11.3  Attachments; 

DA  Form  3645 
DA  Form  3645-1 
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TASK  12A 


12.1.1  Situation! 

DA  Form  2765-1 's  have  been  prepared  for  some  excess 
items  (still  in  depot-pack)  that  have  been  turned  in  to 
DRMO.  The  document  number  for  the  turn-in  is  0331-0006. 


12.1.2  Requirements; 

Post  the  DRMO  turn-in  to  the  Property  Book. 

Once  the  DRMO  turn-in  is  posted,  return  to  the 

Adjustment  Transactions  Menu. 

12.1.3  Attachments! 

DA  Form  2765-1 's 


12.2.1  Situation; 

Several  GIF  items  have  been  lost,  damaged  or  destroyed 
and  have  been  accounted  for  on  a  Statement  of  Charges. 


12.2.2  Recruirements; 

Begin  posting  the  Statement  of  Charges  to  adjust  the 
on-hand  quantities  on  the  Property  Book.  Once  you  have 
input  all  the  NSNs  and  quantities  stop  work  prior  to  moving 
to  the  Total  Price  block. 

Turn  page  for  instructions  on  the  continuation  of  this 
situation. 


12.2.3  Attachments; 

DD  Form  362  (Statement  of  Charges) 
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12.2.4  Situation; 


At  this  time  you  have  just  been  informed  by  your 
supervisor  that  there  was  a  mistake  on  the  document  and  that 
instead  of  Boots  Hot  Weather  8430-00-141-0791  1  PR  being 
lost,  it  was  in  fact  Boots  Hot  Weather  8430-00-141-0796  2  PR 
that  were  lost  and  the  new  Total  Price  of  the  Statement  of 
Charges  is  $72.47. 

12.2.5. Requirements ; 

With  this  information,  finish  posting  the  Statement  of 
Charges . 

Once  the  Statement  of  Charges  is  completed,  return  to 

the  Adjustment  Transactions  Menu. 


12.2.6  Attachments; 

DD  Foirm  362  (Statement  of  Charges) 


12.3.1  Situation; 

A  pair  of  Boots /  Cold  Weather  listed  as  lost  on  a 
Report  of  Survey  has  been  recovered,  and  the  original  R/S 
document  needs  to  be  adjusted  to  exclude  the  recovered  item. 

Recovered  NSNr  8430-00-823-7041 


12.3.2  Requirements; 

Modify  the  Report  of  Survey  to  exclude  the  recovered 
pair  of  Cold  Weather  Boots. 

Once  the  Report  of  Survey  has  been  modified,  return  to 

the  CIF  Main  Menu. 

12.3.3  Attachments; 

DA  Form  4697  (Report  of  Survey) 
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TASK  12B 


12.1.1  Situation: 

DA  Form  2765-1 's  have  been  prepared  for  some  non¬ 
reparable  items  that  have  been  turned  in  to  DRMO.  The 
document  number  for  the  turn-in  is  0331-0007. 


12.1.2  Requirements; 

Post  the  DRMO  turn  in  to  the  Property  Book. 

Once  the  DRMO  turn-in  is  posted,  return  to  the 

Adjustment  Transactions  Menu. 

12.1.3  Attachments! 

DA  Form  2765-1 's 


12.2.1  Situation: 


Several  GIF  items  have  been  lost,  damaged  or  destroyed 
and  have  been  accounted  for  on  a  Statement  of  Charges. 


12.2.2  Requirements; 

Begin  posting  the  Statement  of  Charges  to  adjust  the 
on-hand  quantities  on  the  Property  Book.  Once  you  have 
input  all  the  NSNs  and  quantities  stop  work  prior  to  moving 
to  the  Total  Price  block. 

Turn  page  for  instructions  on  the  continuation  of  this 
situation. 


12.2.3  Attachments ; 

DD  Form  362  (Statement  of  Charges) 
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12.2.4  Situation; 


At  this  time  you  have  just  been  informed  by  your 
supervisor  that  there  was  a  mistake  on  the  document  and  that 
instead  of  Bag  Barracks  Ctn  OG  1  EA  being  lost  it  was  1  EA 
Bag  Duffel  8465-01-117-8699  that  was  lost  and  the  new  Total 
Price  of  the  Statement  of  Charges  is  $54.20. 


12.2.5  Recmirements ; 

With  this  information,  finish  posting  the  Statement  of 
Charges . 

Once  the  Statement  of  Charges  is  completed,  return  to 
the  Adjustment  Transactions  Menu. 


12.2.6  Attachments: 

DD  Form  362  (Statement  of  charges) 


12.3.1  Situation: 

A  Helmet/  6TNP  Parachutist  listed  as  lost  on  a  Report 
of  Suirvey  has  been  recovered,  and  the  original  R/S  document 
needs  to  be  adjusted  to  exclude  the  recovered  item. 

Recovered  MSNi  8470-01-092-7528 


12.3.2  Requirements: 

Modify  the  Report  of  Survey  to  exclude  the  recovered 

Helmet/  GTNP  Parachutist. 

Once  the  Report  of  Survey  has  been  modified,  return  to 
the  CIF  Main  Menu. 


12.3.3  Attachments: 

DA  Form  4697  (Report  of  Survey) 
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TASK  13A 


13.1.3.  Svtui'  ‘  on! 


The  following  items  need  to  be  ordered  to  replenish 
stock. 


NSN 

OTY 

8405-00-782-2121 

10 

8465-00-001-6477 

200 

8430-01-021-5978 

100 

8460-00-606-8366 

20 

UI  PRI  Docvunent  Niimber 

EA  13  _ 

EA  06  _ 

PR  13  _ 

EA  06 


13.1.2  Requirements; 


Generate  supply  requisitions  (Die  AOA  or  AOE)  for  the 
items  and  write  the  computer-assigned  document  numbers  in 
the  right-hand  column. 


Once  the  requisitions  are  completed,  return  to  the 

Document  Register  Actions  Menu. 


13.2.1  situation; 


The  following  item  needs  to  be  ordered  to  replenish 
stock. 

You  are  not  sure  of  the  correct  NSN  because  the  last  digit 
is  unreadable  but  you  know  that  the  item  is  a  case  First  Aid 
Kit  which  the  CIF  stocks.  The  last  digit  is  either  a  9  or  a 

4. 

NSN  OTY  PI  PRI  Document  Number 

8465-00-935-681X  250  EA  06 


13.2.2  Requirements: 

Generate  a  supply  requisition  (Die  AOA)  for  the  item. 
Try  inputting  the  NSN  with  9  as  the  last  digit  first. 

Once  you  realize  that  the  NSN  is  incorrect  (because  the 
one  yo’  input  is  not  on  the  stock  Master  Record) ,  then  try 
again  uo  generate  the  appropriate  supply  requisition  using  4 
as  the  last  digit  of  the  NSN. 

Write  the  computer-assigned  document  number  in  the 
indicated  column  above. 

Once  the  requisition  is  completed,  return  to  the 

Document  Register  Actions  Menu. 
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13.3.1  Situation; 


A  supply  requisition  created  earlier  in  the  day  needs 
to  be  reduced  in  order  quantity.  The  requisition  for  the 
item  needs  to  be  modified  now,  before  it  is  passed  forward 
to  higher  supply  levels  at  close  of  business. 


Document  Humber _ 0331-0003 _ 

Quantity  to  Cancel _ 4  EA _ 

13.3.2  Requirements: 

Post  a  modification  to  the  document. 

Once  the  supply  requisition  has  been  modified,  return 
to  the  CIF  Main  Menu. 
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TASK  13B 


13.1.1  Situation; 

The  following  items  need  to  be  ordered  to  replenish 
stock. 

WSN 

8405-00-001-8026 
8415-00-782-2887 
8440-00-782-2173 
8410-01-170-7556 


13.1.2  Requirements; 

Generate  supply  requisitions  (DIG  AOA  or  AOE)  for  the 
items  and  write  the  computer-assigned  document  numbers  in 
the  right-hand  column. 

Once  the  requisitions  are  completed,  return  to  the 

Document  Register  Actions  Menu. 


OTY 

UI 

PRI 

100 

PR 

13 

60 

EA 

06 

150 

PR 

06 

30 

PR 

13 

Document  Number 


13.2.1  Situation; 

The  following  item  needs  to  be  ordered  to  replenish 
stock.  You  are  not  sure  of  the  correct  NSN  because  the  last 


digit  is  unreadable 
Balaclava  which  the 
4  or  a  9. 

but 

GIF 

you  know 
stocks . 

that 
The  , 

the  item  is  a  Hood 
last  digit  is  either  a 

NSN 

OTY 

UI 

PRI  Document  Number 

8415-01-111-115X 

150 

EA 

06 

13.2.2  Requirements: 


Generate  a  supply  requisition  (DIG  AOA)  for  the  item. 
Try  inputting  the  NSN  with  4  as  the  last  digit  first. 

Once  you  realize  that  the  NSN  is  incorrect  (because  the 
one  you  input  is  not  on  the  Stock  Master  Record) ,  then  try 
again  to  generate  the  appropriate  supply  requisition  using  9 
as  the  last  digit  of  the  NSN. 

Write  the  computer-assigned  document  number  in  the 
indicated  column  above. 

Once  the  requisition  is  completed,  return  to  the 

Document  Register  Actions  Menu. 
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13.3.1  Situation; 


An  item  ordered  earlier  in  the  day  is  no  longer 
required.  The  requisition  for  the  item  needs  to  be 
cancelled  now,  before  it  is  passed  forward  to  higher  supply 
levels  at  close  of  business. 


Document  Number _ 0331-0004 _ 

Quantity  to  Cancel  20  EA _ 

13.3.2  Recmirements; 

Post  a  local  cancellation  to  the  document. 

Once  the  supply  requisition  has  been  cancelled,  return 
to  the  CIF  Main  Menu. 
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TASK  15A 


15.1.1  Situation; 

A  new  soldier  has  arrived  at  your  desk  carrying  a 
Worksheet  which  has  been  annotated  with  the  sizes  and 
quantities  of  items  issued  to  her.  Behind  her  is  a  long 
line  of  soldiers  who  are  also  waiting  to  see  you  with  their 
issue  paperwork. 

15.1.2  Requirements: 

Begin  posting  the  sizes  and  quantities  to  the  soldier's 
record  in  the  fastest  wav  possible  and  once  you  are  ^  LIN 
F54817  and  have  entered  the  quantity  issued  stop  work. 

Turn  the  page  for  instructions  on  the  continuation  of 
this  situation. 

15.1.3  Attachments; 


Worksheet  (1st  Half) 


15.1.4. Situation ; 

While  you  were  posting  the  issue,  the  soldier  noticed 
that  both  pairs  of  Boots  Hot  Weather  that  were  issued  to  her 
were  in  fact  defective  and  when  she  went  to  get  serviceable 
ones  off  the  line  she  came  back  with  two  pairs  of  a  new  size 

-  4-1/2  Regular. 

15.1.5  Requirements; 

Make  a  correction  to  her  record  on  the  screen  and 
complete  her  issue  as  quickly  as  possible. 

Once  you  have  completed  posting  the  issue,  return  to 

the  Clothing  Issues  Process  Menu. 

15,1.3  Attachments; 

Worksheet  (1st  Half) 

Worksheet  (2nd  Half) 
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IS » 2.1  Situation 


A  soldier  has  just  been  issued  items  which  previously 
were  not  available  for  issue  at  the  time  of  his  initial 
issue. 


SSN; _ 405-78-3380 _ 

LINS  Issued;  C07440  &  F30391 

QTYs  Issued;  2 _ 1 


15.2.2  Recmirements; 

Post  the  issue  of  these  items  to  the  soldier's  clothing 
record. 

Once  the  issue  of  due-out  items  has  been  posted,  return 
to  the  GIF  Main  Menu. 


152 


TASK  15B 


15.1.1  Situation; 

A  new  soldier  has  arrived  at  your  desk  carrying  a 
Worksheet  which  has  been  annotated  with  the  sizes  and 
quantities  of  items  issued  to  her.  Behind  her  is  a  long 
line  of  soldiers  who  are  also  waiting  to  see  you  with  their 
issue  paperwork. 

15.1.2  Recmirements! 

Begin  posting  the  sizes  and  quantities  to  the  soldier's 
record  in  the  fastest  vav  possible  and  once  you  are  ^  LIN 
fr06645  and  have  entered  the  quantity  issued  stop  work. 

Turn  the  page  for  instructions  on  the  continuation  of 
this  situation. 

.1.3  Attachments; 

Worksheet  (1st  Half) 

15.1.4  Situation; 

While  you  were  posting  the  issue,  the  soldier  noticed 
that  the  Overshoes,  Vinyl  OG  that  were  issued  to  her  were  in 
fact  the  wrong  size  and  when  she  went  to  get  serviceable 
ones  off  the  line  she  came  back  with  Overshoes  of  a  new  size 

-  ft. 

15.1.5  Requirements; 

Make  a  correction  to  her  record  on  the  screen  and 
complete  her  issue  as  quickly  as  possible. 

Once  you  have  completed  posting  the  issue,  return  to 

the  Clothing  Issues  Process  Menu. 

15.1.6  Attachments; 

Worksheet  (2nd  Half) 
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15.2.1  Situation: 


A  soldier  has  just  been  issued  items  which  previously 
were  not  available  for  issue  at  the  time  of  his  initial 
issue. 


SSN: _ 583-42-7075 _ 

LIN  Issued: _ Dliei2  fc  L00210 

QTY  Issued: _ i _ i 


15.2.2  Recmirements: 

Post  the  issue  of  these  items  to  the  soldier's  clothing 
record. 

Once  the  issue  of  due-out  items  has  been  posted,  return 
to  the  GIF  Main  Menu. 
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APPENDIX  6 

INSTRUCTIONS  TO  TEST  PARTICIPANTS 


155 


IMSTROCTIONS  TO  TEST  PARTICIPAWTS 


1.  Purpose  of  Teatina.  The  purpose  of  the  testing  is  to  determine 
to  what  extent  Dynamic  HELP  is  useful  to  a  supply  clerk  using  the 
CIF  program.  The  CIF  software  you  will  be  using  today  has  many 
confusing  aspects  to  it,  and  we  have  NOT  attempted  to  fix  it..  we 
are  trying  to  see  how  well  Dynamic  HELP  can  get  you  around  the 
confusing  aspects  in  the  software  so  that  you  can  perform  your 
job  more  quickly  and  accurately. 


2.  nyTuuB^c  HELP.  A  Dynamic  HELP  Message  is  a  message  based  upon 
where  you  are  in  the  program  and  what  state  the  program  is  in  at 
the  time  the  HELP  message  is  called.  The  message  is  specific  to 
what  kinds  of  questions  the  user  might  ask  at  that  particular 
point  in  the  program.  During  half  of  the  test  you  will  be 
permitted  to  use  Dynamic  HELP.  You  are  stronol v  encouraged  to 
use  Dynamic  HELP  whenever  you  are  unsure  or  confused  while  in  any 
data-entry  or  menu  selection  screen. 


3.  Sequence  of  Testing.  There  are  3  phases  of  testing. 

A.  Phase  I  is  the  Pre-Test  Survey  where  personal  data  and 
information  on  your  supply  experience,  CIF  work  experience,  and 
computer  terminal  experience  will  be  collected. 

B.  Phase  II  -  Task  Testing. 

1)  Task  Performance.  You  will  perform  8  tasks  on  the 
computer.  For  4  of  the  tasks  you  may  use  the  Dynamic  HELP  Key 
(P4)  anytime.  For  the  other  4  tasks  you  may  not  use  Dynamic 
HELP.  Those  tasks  which  you  mav  use  Dynamic  HELP  with  are 
clearly  marked.  In  order  to  perform  the  8  tasks,  you  will  need  a 
basic  understanding  of  the  following  forms: 

DA  Form  2765-1  (Request  for  Issue  or  Turnin) 

DA  Form  3645/3645-1  (Clothing  Record) 

DA  Form  2064 (Document  Register  for  Supply  Actions) 
DA  Form  4697  (Report  of  Survey) 

DD  Form  362  (Sracement  of  Charges) 

If  you  are  not  familiar  with  all  of  the  above  documents,  please 
inform  the  System  Administrator.  During  testing,  you  may  use  a 
pencil  (provided)  to  write  on  the  supply  forms  that  accompany  the 
task  instructions.  You  may  use  a  non-permanent  marker  (also 
provided)  to  annotate  the  task  instruction  paperwork. 

2)  Post-Task  Surveys.  After  each  task,  you  will 
complete  a  post-task  survey  to  describe  any  interface 
difficulties  you  encountered  while  performing  the  task. 
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^1  Lunch.  You  will  complete  4  tasks  and  the 
corresponding  post-task  surveys  prior  to  the  lunchbreak.  The 
System  Administrator  will  tell  you  when  to  return  from  lunch  to 
resume  testing. 

C.  Phase  III  is  the  Post-Test  Survey  which  consists  of 
general  questions  about  Dynamic  HELP. 


4 .  Questions  Purina  Testing. 

A.  Surveys .  While  completing  any  survey,  you  may  ask  the 
System  Administrator  to  clarify  any  unclear  question  meanings  or 
response  instructions. 

B.  Task  Performance.  Perform  each  task  as  accurately  and 
rapidly  as  the  software  will  let  you.  This  is  a  test  of  the 
software,  not  of  you.  If  the  task  itself  is  confusing,  ask  the 
System  Administrator  for  clarification  immediately.  But  if  how 
to  use  the  computer  to  do  the  task  is  unclear  (in  other  words, 
you  are  stuck)  don't  stop  work,  try  the  following: 

1)  osa  Dynamic  HELP  (if  allowed) 

2)  Trial  and  error  (experimenting) 

3)  Search  the  screen  for  Instructions 

Ask  the  System  Administrator  for  computer-related  assistance  only 
if  you  think  it  would  take  a  whole  minute  or  more  to  discover  how 
to  proceed.  Tell  the  System  Administrator  you  have  reached  a 
Dead-End  (where  you  would  normally  have  to  get  out  the  manuals  or 
ask  someone  what  to  do) .  At  that  point,  the  System  Administrator 
will  do  the  minimum  necessary  to  assist  you  in  resuming  task 
performance. 


5.  .mtSgruptiqp- 

A.  Machine  Shutdown.  If  there  is  an  interruption  or  machine 
breakdown  please  notify  the  System  Administrator  immediately.  If 
task  data  is  lost,  you  will  be  asked  to  restart  the  task  from  the 
beginning . 

B.  other  Interruptions.  If  there  is  an  interruption  due  to 
external  forces,  such  as  a  fire  drill  or  a  latrine  break,  please 
notify  the  System  Administrator.  Leave  data  undisturbed.  The 
System  Administrator  will  record  the  time  of  the  interruption  and 
resumption,  when  you  resume  the  task  session. 
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6.  Program  Problems.  While  using  this  program,  you  may 
experience  a  couple  of  “odd”  occurrences.  The  first  type  of 
problem  is  when  you  hit  a  key  while  in  the  middle  of  a  task,  and 
the  program  sends  you  all  the  way  out  to  the  CIF  Main  Menu.  The 
second  type  of  problem  is  when  you  are  at  the  end  of  a  screen  and 
you  press  ESC  or  some  other  key  to  process  the  information,  and 
the  program  jumps  the  cursor  back  up  to  the  beginning  of  the 
screen  or  page.  If  either  of  these  two  problems  occur  while  you 
are  working,  notify  the  System  Administrator  immediately.  Also, 
if  any  other  unexpected  problems  occur,  inform  the  System 
Administrator . 

7.  Assignment  of  OSER  ID.  A  USERID  will  be  used  to  identify  you 

throughout  testing  and  post-testing  data  analysis,  your  USERID 
is  _  _ . 
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1.  Purpose  and  Structure  of  Experiment.  The  purpose  of  this 
experiment  is  to  determine  whether  and  to  what  extent  Dynamic 
HELP  is  helpful  to  a  supply  clerk  using  the  CIF  Program  to 
perform  CIF  functions.  An  incidental  purpose  is  to  gather 
diagnostic  information  to  indicate  relationships  between 
different  kinds  or  parts  of  Dynamic  HELP  messages  and  different 
user  situations  or  confusions.  The  experiment  gathers 
information  about  the  user  in  a  Pre-Test  Survey,  gathers 
observations  of  user  actions  and  task  results  in  the  user's 
performance  of  ten  CIF  tasks  with  and  without  Dynamic  HELP,  and 
gathers  user  opinions  in  a  Post-Test  Survey.  The  user  is  assumed 
to  be  attempting  to  perform  tasks  as  accurately  and  quickly  as 
the  CIF  software  will  allow. 

For  a  given  task,  or  a  given  part  of  a  task,  the  evidence 
being  gathered  anout  Dynamic  HELP'S  usefulness  is 


A.  The  user's  frequency  of  using  it  when  available. 

B.  The  avoidance  of  Dead-Ends. 

C.  Reduction  in  measures  of  effort  and  confusion,  such  as 
elapsed  time,  number  of  departures  from  a  direct  path,  number  of 
corrections  or  re-entries,  number  of  keystrokes,  etc. 

D.  Accuracy  of  the  task  result,  as  measured  in  correctness 
of  the  data  changes  and  reports  that  the  task  generated. 

E.  User  opinion  as  to  whether  or  to  what  extent  Dynamic  HELP 
was  useful. 


2.  Purpose  of  Handbook.  The  purpose  of  this  handbook  is  to  set 
forth  guidelines  and  procedures  to  allow  for  administration  and 
control  of  the  User  Testing  of  the  converted  CIF  software  at 
Information  Systems  Command's  Software  Development  center  at  Fort 
Gillem,  Georgia. 


3 .  The  Expt ^ineutal  Sequence . 


A.  Daily  Schedule. 


Initial  Briefing 
Pre  -Test  Survey 
Testing:  2  users  per  day 
Task  Testing  (5  tasks)  k 
Post-Task  Surveys 
Lunch  Break 

Task  Testing  (5  tasks)  & 
Post-Task  Surveys 
Post-Test  Survey 
Recheck/Release 


0800-0830 

0830-0900 

0900-1130 

1130-1330 
1-  0-1600 

1600-161S 

1615-1630 


B,  Daily  Starring  Procedures.  Each  morning,  the  following 
actions  will  be  pe*.  ormed  prior  to  the  start  of  testing  (0800 
hrs)  ; 


1)  Logon  to  system  (2  Wyse  terminals) . 

2)  Database  reinitialization. 

3)  Ensure  testing  materials  are  prepared: 

a.  Instructions  to  Test  Participants 

b .  Pre-Testing  Survey 

c.  Task  Notebooks 

i.  Sequence  of  a.m.  tasks 

ii.  Availability  of  task  forms 
iii-  Post -Task  Surveys 

d.  Observation  Worksheets 

e .  Post-Test  Surveys 

f.  Pencils,  Grease  Pencils  and  Cleaner 

g.  Notebook  with  p.m.  tasks 

4)  Diagnostic  check  of  tasks  by  System  Administrator. 

5)  Call  up  CIF  Main  Menu  on  both  Wyse  terminals. 

C.  Initial  Briefing  to  Participants. 

1)  Test  participants  will  be  briefed  on  or  will  read 
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the  Initial  Instructions  prior  to  the  Pre-Test  Survey.  See 
Instructions  to  Test  Participants  (Enclosure  1)  for  specific 
content. 


D.  Pre-Test  Survey. 

1)  Description.  The  Pre-Test  Survey  collects  personal 
data  and  information  on  supply  experience,  CIF  work  experience, 
and  computer  terminal  experience.  It  is  to  be  completed  prior 
to  the  beginning  of  the  Task  Sessions.  See  Enclosure  2. 

2)  Participant  Questions.  While  completing  the  survey, 
test  participants  may  ask  the  System  Administrator  to  clarify  any 
unci ear  question  meanings  or  response  instructions. 

3)  Materials  Required.  Each  test  participant  will 
receive  the  pre-test  survey  and  a  pencil  with  which  to  complete 
it. 


E.  Task  Testing. 

1)  Task  Performance. 

a.  Description  of  Tasks.  Each  test  participant 
will  perform  a  total  of  10  tasks,  5  with  Dynamic  HELP  disabled 
and  5  with  Dynamic  HEIJ>  enabled.  See  Enclosure  3  for  copies  of 
the  10  base  tasks.  The  System  Administrator  will  ensure  task 
sequences  are  assigned  as  specified  in  paragraph  4.  The  tasks 
will  be  within  the  following  submenu  areas  of  the  CIF  Program; 


Clothing  Issue  Processes 
Clothing  Turnin  Processes 
Document  Register  Actions 
Adjustment  Transactions 


b.  Physical  Control  of  Dynamic  HELP  Key.  For 
those  task  sessions  where  the  Dynamic  HELP  key  is  not  allowed  to 
be  used,  the  key  will  be  physically  covered  to  prevent  the  test 
participant  from  accidentally  using  it  during  task  performance. 

c.  Session  Sequence. 

i.  Header  Call 

ii.  Task  Session 

iii.  Closure  Call 

Dead-End  Encounters 
Remarks 
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d.  Dead  End  Encounters. 


i.  Definition.  A  Dead-End  is  defined  as  ttie 
^jxnt  a  test  participant  reaches  where  he/she  realizes  that  it 
would  take  at  least  one  minute  more  to  discover  how  to  proceed 
with  or  finish  a  task  because  of  confusion  with  the  computer 
interface. 


ii.  Procedures.  The  test  participants  will 
be  thoroughly  briefed  on  the  definition  and  procedures  of  a  Dead- 
End.  When  a  test  participant  declares  to  be  at  a  Dead-End  Point, 
the  System  Administrator  will  note  the  Task  ID,  USER  ID,  Subtask 
ID,  Screen/Field  ID  and  other  descriptive  information  on  the 
Observation  Worksheet.  Then  the  System  Administrator  will  do  the 
minimum  actions  necessary  to  assist  the  test  participant  in 
resuming  and  finishing  the  task  session.  The  System 
Administrator  will  enter  the  Dead-End  Encounter  information  at 
the  Closure  Call  for  the  task  session  in  which  it  occurred. 

e.  Allowable  Questions.  While  engaged  in  a  task 
session,  test  participants  may  ask  the  System  Administrator  to 
clarify  any  confusion  on  the  intention  of  a  task  itself,  but  may 
not  ask  for  clarification  on  how  to  use  the  computer  to  perform 
the  task. 


f.  Observations  by  System  Administrator. 

i.  Dead  Ends.  See  Paragraph  3.£.l)d. 

ii.  Interrupts.  See  Paragraph  5A. 

iii.  Other  Observations.  During  the  task 
sessions,  the  System  Administrator  will  make  note  of  any  unusual 
behavior  by  test  participants  onto  the  Observation  Worksheet  (See 
Enclosure  4)  and  will  enter  any  notes  made  during  the  session 
under  Remarks  at  the  Closure  Call.  Also,  if  a  participant 
consistently  exhibits  any  behavior  which  might  impact  on  the  test 
results,  the  System  Administrator  should  record  the  information. 
For  instance,  if  the  user  appeared  to  be  ill,  a  record  should  be 
made  of  the  user  and  the  illness  observation  so  that  the 
information  is  available  during  data  analysis. 

g.  Materials  Required.  Each  test  participant  will 
receive  a  notebook  with  5  sequenced  tasks  prior  to  the  beginning 
of  testing  (pre-lunch  and  post-lunch) .  The  notebook  will  contain 
all  necessary  references  or  paperwork  for  each  task.  The  test 
participant  will  also  be  provided  with  a  non-permanent  marker,  as 
some  tasks  require  the  participant  to  write  down  document 
numbers,  etc.  on  task  paperwork.  Also,  the  test  participant  will 
be  given  a  pencil  which  may  be  used  to  mark  on  accompanying 
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supply  forms  such  as  DA  Form  3645s,  etc.  to  assist  in  task 
completion. 

2)  Post-Task  Surveys. 

a.  Description.  The  Post-Task  Surveys  collect 
task-specific  information  regarding  the  use  of  Dynamic  HELP  and 
identities  those  areas  of  the  task  session  where  the  user  was 
confused  by  the  interface.  There  is  a  HELP  and  Non-HELP  version 
of  the  survey  for  each  task  type.  A  Post-Task  Survey  will  be 
completed  by  the  test  participant  after  each  task  session.  See 
Enclosure  5  for  copies  of  the  post-task  surveys. 

b.  Participant  Questions.  While  completing  the 
survey,  test  participants  may  ask  the  System  Administrator  to 
clarify  any  unclear  guestion  meanings  or  response  instructions. 

c.  Materials  Reguired.  Each  test  participant  will 
receive  the  applicable  post-task  survey  and  a  pencil  with  which 
to  complete  it  at  the  time  of  the  Closure  Call. 

3)  Lunch.  Each  test  participant  vlll  be  released  to 

eat  lunch  at  the  discretion  of  the  Sys-?-!  Acc ’■  istrator .  In 
general,  test  participants  will  have  a  treak  after  the 

first  5  tasks  and  post-task  surveys.  Tht  lu  ,  rh  break  will  be  no 
longer  than  2  hours  in  length.  During  the  lunch  break,  the 
System  Administrator  will  prepare  task  notebooks  for  post-lunch 
testing  (ie,  erase  grease  pencil  markings  and  rearrange  tasks) 
and  may  begin  entering  pre-survey  data  into  USER  database  table. 
Also,  the  System  Administrator  may  view  the  Task  History  Tables 
to  ensure  data  is  being  posted  correctly. 


F.  Post -Test  Surveys. 

a.  Description.  The  Post-Test  Survey  collects 
general  user  opinions  on  Dynamic  HEIR.  It  will  be  completed  by 
each  test  participant  after  he/ she  has  completed  all  task 
sessions.  See  Enclosure  6  for  a  copy  of  the  post-test  survey. 

b.  Participant  Questions.  While  completing  the 
survey,  test  participants  may  ask  the  System  Administrator  to 
clarify  any  unclear  guestion  meanings  or  response  instructions. 

c.  Materials  Required.  Each  test  participant  will 
receive  the  post-test  survey  and  a  pencil  with  which  to  complete 
it. 


G.  Release  of  Participants.  Test  participants  will  be 
released  from  the  experiment  by  the  System  Administrator  after  a 
final  check  of  survey  paperwork. 
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H.  Daily  Shutdown  Procedures.  Each  evening,  the  following 
<.  wions  will  be  performed  after  the  conclusion  of  testing  (1700 
hrs)  : 

1)  Collect  all  surveys  and  task  printouts. 

2)  Run  Session  Closure  Reports  (?) 


3)  Reinitialize  database. 

4)  Prepare  next  day's  task  sequences,  surveys, etc. 

5)  Logout  of  system  (2  Wyse  terminals) . 


4  .  Task  Secruenee  Aasioiunents . 

A.  Assignment  Procedure.  There  are  five  task  types .  There 
are  two  versions  of  each  task  type  and  either  version  can  be 
performed  with  or  without  Dynamic  HELP.  To  ensure  that  a 
sufficient  amount  of  time  has  transpired  between  performance  of 
"like"  tasks,  one  version  of  each  task  type  will  be  completed 
prior  to  the  user's  lunch  break.  The  remaining  version  of  each 
task  type  will  be  completed  after  a  two-hour  lunch.  At  most  two 
users  will  undergo  testing  during  a  day.  If  one  user  undergoes 
testing,  he/ she  is  considered  the  odd-numbered  user.  If  a  second 
participant  also  undergoes  testing  the  same  day,  he/ she  is 
considered  the  even-numbered  user 

The  orders  of  task  type  performance  before  lunch  as  well  as 
well  as  after  lunch  are  random.  The  version  of  each  task  type 
performed  before  lunch  is  random  for  the  odd-numbered  users  (the 
even-numbered  user  will  perform  the  other  task  type  versions 
before  lunch  to  ensure  database  integrity) .  During  the  lunch 
break,  the  database  will  be  reinitialized.  Then,  the  users  will 
perform  the  remaining  version  of  the  tasks  (the  task  type 
versions  they  have  not  yet  performed)  after  lunch.  The 
assignment  of  HELP  and  Non-HELP  within  the  task  sequence  before 
lunch  is  random  while  after  lunch  versions  of  each  task  type  will 
receive  the  opposite  assignment  of  HELP  and  Non-HELP.  For 
instance,  given  that  the  task  types  are  numbered  1-5  and  the  two 
versions  are  named  A  and  B,  with  a  random  number  generator  one 
could  assign  the  following  before  lunch  task  type  sequence  for 
participant  (fl: 


5  3  4  1  2 

and  an  after  lunch  sequence  of: 

5  12  4  3 

This  randomization  will  nullify  any  "anticipation"  effects  of 
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users  performing  the  same  sequence  of  task  types  after  lunch. 
Random  assignment  of  task  type  versions  could  then  yield: 
Before  Lunch:  5A  3A  4A  IB  2B 

and 

After  Lunch:  5B  lA  2A  4B  3B 

Then,  random  assignment  of  HELP  and  Non-HELP  could  yield: 
Before  Lunch:  5AH  3AN  4AH  IBN  2BH 

and 

After  Lunch:  5BN  lAH  2 AN  4 BN  3BH 

where  "5AH"  corresponds  to  Task  Type  5,  Version  A,  to  be 
performed  with  Dynamic  HELP. 


fi.  Actual  Assignment  for  Experiment.  The  same  logic  was 
used  to  generate  random  task  performance  sequences  for  all  20-25 
test  participants .  Using  Table  XV  from  Hines  and  Montgomery's 
Probability  and  Statistics  in  Engineering  and  Management  Science 
and  given  that  the  task  types  are  numbered  11-15,  versions  are 
designated  A  and  B,  HELP  is  designated  as  H  and  Non-HELP  is 
designated  as  N,  the  resultant  task  perfomance  sequence  was: 


USER# 

1 

! 

2M 

1 

15AH 

13AN 

14AH 

llBN 

12BH  I 

ISBN 

llAH 

12AN 

14BN 

13BH 

2 

12AH 

llAH 

13BH 

ISBN 

14  BH  [ 

13AN 

ISAH 

14AN 

12  BN 

llBN 

3 

14AK 

13AN 

11  AH 

15BH 

12AN  ] 

13BH 

14BH 

ISAN 

12BH 

llBN 

4 

12BH 

IIBH 

13BN 

14BH 

15AH  1 

llAN 

13AH 

ISBN 

14AN 

12AN 

5 

13BN 

15BN 

14  BN 

llBN 

12AN  ! 

14AH 

llAH 

15AH 

13AH 

12BH 

6 

15AH 

llAN 

14AH 

13AN 

12  BH  j 

ISBN 

IIBH 

13BH 

12AN 

14BN 

7 

15BH 

13AN 

IIBH 

12AH 

14  AN  [ 

15AN 

llAN 

14BH 

12  BN 

13BH 

8 

14  BN 

12  BH 

15AH 

13BN 

llAH  1 

12AN 

llBN 

ISBN 

13AH 

14AH 

9 

14  BN 

15BH 

12BN 

13AH 

iiAN  : 

12AH 

ISAN 

14AH 

13BN 

IIBH 

10 

15AH 

IIBN 

12AK 

13BH 

14AH  1 

13AN 

ISBN 

12BN 

14  BN 

llAK 

11 

14AN 

13AH 

12BN 

ISBN 

IIBH  1 

13BN 

llAN 

12AH 

ISAH 

14BH 

12 

15AN 

14BN 

13BN 

llAH 

12AN  ] 

15BH 

llBN 

13AH 

12  BH 

14AH 

13 

13  BN 

14BN 

12AH 

15AN 

llAH  1 

13AH 

14AH 

12BN 

llBN 

15BH 

14 

12BN 

15BN 

13AN 

14AK 

IIBH 

llAN 

13BH 

12AH 

14BN 

ISAH 
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USER* 

1 

1 

PM 

If 

12BN 

14AH 

IIBH 

13AH 

15BH  1 

13BN 

14  BN 

15AN 

12  AH 

llAN 

16 

14  BN 

13BH 

12AtJ 

15  AH 

llAH  1 

12BH 

14AH 

13  AN 

15BH 

llBN 

17 

12BN 

15AH 

14  BH 

llBN 

13BN  1 

14AN 

llAH 

12AH 

13AH 

15BN 

18 

12AH 

llAH 

15BH 

14AN 

13  AN  ; 

13BH 

14BH 

15AN 

llBN 

12BN 

19 

15BN 

14AH 

llAH 

13AH 

12  BN  1 

llBN 

13BH 

15AH 

12AH 

14BN 

20 

12AH 

11  BN 

14BH 

15  AN 

13BH  ; 

14AN 

15BH 

12  BN 

13  AN 

llAH 

21 

13AH 

llBN 

12BH 

15BH 

14AH  1 

llAH 

13BN 

15AN 

12AN 

14BN 

22 

15AN 

llAN 

12AN 

13BH 

14BN  1 

14AH 

12BH 

15BH 

IIBH 

13AN 

23 

14AH 

llAH 

15BH 

12BN 

13AH  1 

15AN 

llBN 

13BN 

12AH 

14BN 

24 

12AH 

llBN 

13BN 

15  AN 

14BN  1 

14AH 

13AH 

llAH 

12  BN 

15BH 

25 

IIBN 

14  BH 

13AH 

15BH 

12BH  1 

12AN 

14AN 

llAH 

13BN 

15AN 

26 

15AH 

13BN 

llAN 

12  AH 

14  BN  j 

15BN 

IIBH 

14AN 

12BN 

13AK 

Note  that  the  odd-numbered  users  had  their  task  type  versions 
randomly  assigned  in  the  a.m.  and  therefore  their  p.m.  task  type 
versions  were  opposite  those  of  the  a.m.  Also,  the  even-numbered 
users'  task  type  versions  are  opposite  those  of  the  preceding 
odd-numbered  user  both  in  the  a.m.  and  p.m.  This  will  ensure 
database  integrity  throughout  testing.  Note:  This  sequence 
assignment  assumes  that  two  participants  will  be  present  each  day 
with  the  possible  exception  of  one  day. 


5.  special  Instructions. 

A.  Task  Interruption. 

1)  Machine  Shutdown.  If  at  any  time  during  a  task 
session,  there  is  an  interruption  due  to  machine  breakdown  where 
task  performance  data  is  lost,  the  task  must  be  restarted  after 
database  reinitialization.  Any  Task  History  Table  or  Session 
File  entries  from  the  earlier  interrupted  task  will  be  rewritten 
to  a  separate  "interruption  data"  table  and  will  be  available  for 
analysis  if  necessary. 

2)  Other  Interruptions.  If  at  any  time  during  a  task 
session  there  is  an  interruption  due  to  external  forces,  such  as 
a  fire  drill  or  a  latrine  break,  and  the  task  performance  data  is 
undisturbed,  then  the  system  Administrator  must  record  the  time 
of  the  interruption.  Once  the  interruption  is  concluded  and  the 
task  session  resumes,  the  System  Administrator  must  record  the 
time  of  resumption.  Both  times  should  be  annotated  in  the 
Closure  Call  for  the  task  session. 
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£.  Guidelines  for  Administraror .  Administrator  actions  can 
affect  the  validity  of  the  experiment.  In  particular,  the  System 
Administrator  should  consistently  observe  the  following 
guidelines : 

X)  Do  not  influence  or  aavise  the  user  except  to 
mention  whether  or  not  Dynamic  HELP  is  available  for  the  given 
task. 


2)  Note  any  interruptions  and  record  their  time. 

3)  Note  any  departures  from  usual  assumed  behavior  that 
may  invalidate  results.  If  possible,  ensure  that  the  user  is 
attempting  to  perform  tasks  as  accurately  and  quickly  as  the  GIF 
software  allows.  Prevent,  or  note,  frivolity,  anger,  and  idle- 
curiosity  exploration. 

4)  Make  the  definition  of  Dead-Ends  consistent  and 
known  to  the  user.  Dead-Ends  are  counted  separately  from  elapsed 
time  because  one  Dead-End  could  cause  a  huge  elapsed  time  that 
would  statistically  overwhelm  the  results.  When  a  confusion  is 
believed  to  be  deep  enough  that  a  whole  minute  of  further  effort 
would  be  required  to  proceed,  a  Dead-End  is  declared,  and  the 
user  is  given  the  minimum  personal  help  to  proceed. 

5)  Answer  all  questions  about  supply  procedures  or 
•urveys.  Answer  no  questions  about  computer  procedures. 


Enclosures: 

1  -  Instructions  to  Test  Participants 

2  -  Pre-Test  Survey 

3  -  The  10  Base  Tasks 

A  -  Observation  Worksheet 

5  -  Post-Task  Surveys 

6  -  Post-Test  Survey 
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APPENDIX  8 

SPECIFICATION  FOR  SESSION  HISTORY  DATA  COLLECTION  &  REPORT 
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SPECinCRTION  FOR  SESSION  HISTORY  DATA  COLLECTION  k  REPORT 


1.  In  the  Design  of  the  Evaluation  of  the  converted  GIF  Program 
there  are  ten  separate  tasks  that  participants  will  perform.  A 
MBsion  is  defined  as  a  user's  performance  of  a  single  task. 
During  each  session,  records  will  be  inserted  into  the  Task  History 
Taible  (TET)  for  the  task  and  at  session  closure,  a  Session  Closure 
Report  (SCR)  will  be  generated. 

2.  Task  History  Table  (TJIT)  .  There  will  be  10  TKT's,  each  with  a 
name  that  identifies  its  task.  Queries  from  the  THT  will  be  sorted 
in  increasing  date/ time  order.  Enclosure  l  lists  the  THT  column 
headings  for  each  task.  Criteria  for  insertion  into  the  THT  are 
given  in  Enclosure  2  for  each  task.  Each  THT's  criteria  consists 
of  Key  Collection  Identifications,  other  Collection  Attributes,  and 
Table  Inclusion  Logic. 

3.  Session  Closure  Report  (SCRi  .  The  SCR  is  a  text  file  whose 
filename  will  identify  the  task  and  user.  It  will  consist  of  the 
following  sequence  of  items; 

a)  Header  Stamp  (See  Enclosure  3  for  description) 

b)  "Other  Items"  not  necessarily  available  from  queries 
of  the  THT ( including  quality  checks  and  special  key 
totals) .  Parameters  which  will  then  be  used  by 
queries  to  the  THT  are  denoted  by  ( iparametername ) . 
These  are  also  specified  in  Enclosure  4. 

c)  Results  of  Queries  to  the  TET  and  CIP  DBase. 

Enclosure  4  contains  the  specific  query  logic  in 
pseudo-sql  for  each  task. 

d)  Closure  stamp.  (See  Enclosure  3) 

4 .  The  following  diagram  depicts  the  sequence  of  events  occuring 
within  any  particular  session: 

I  Generate  Session's  Rows  T  ;  Write  Session's  "Other  ; 

in  Task  History  Table  Items"  to  Session  , 

J _ I  i _ Closure  Report _ 


I  Read  Session's  Closure  Report  ! 
i _ for  Quer\'  Parameters _ L 


Query  THT  and  CIF  Database 
Append  Results  to  End  of 
Session's  Closure  Report 
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5.  Please  "create"  the  ten  required  THTs  (ie  establish  column 
headings  and  appropriate  domains) .  For  each  tas)c,  insert  logic 
into  the  CIF  program  modules: 

a)  To  make  insertions  into  the  THTs. 

b)  To  collect  the  Header  Stamp. 

c)  To  collect  on-the-fly  non-query’  items  for  the  SCR. 

d)  To  perform  THT  and  CIF  DBase  queries  for  Session 
Closure  Reports. 

e)  To  collect  the  Closure  Stamp. 

f)  To  assemble  the  Session  Closure  Report,  generate  its 
filename  and  save  it. 

g)  To  cause  the  database  to  be  reinitialized  at  time  of 
Closure. 

6.  In  support  of  the  data  collected  during  user-testing,  there  is 
a  need  for  a  User  Table  which  will  be  indexed  by  User  ID  and  will 
contain  each  user's  name  and  other  data  relevant  to  the  testing 
sessions.  The  User  Table  will  be  created  with  column  names  and 
domains  specified  by  CPT  Wolven  (to  be  provided  at  a  later  date) . 
Actual  data  entry  will  be  performed  by  CPT  Wolven. 
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TASK  HISTORY  TABLE  COLD?P»  HEADINGS 


TASK  11 

(A  Si  B)  (TableManes  TllA  (■  TUB) 

Coltuan  Headina 

Description 

User 

User  Identification  Code 

Date/Time 

Date/Time  Code 

Key 

Key  Identification  Code 

Scr/Fld 

Screen  and  Field  Identification  Code 

LIN 

LIN  Number 

TASK  12  (A  fc  B)  (TableMaaes  T12A  b  T12B) 


Column  Heading 
User 

Date/Time 

Key 

Scr/Fld 


Deseription 

User  Identification  Code 

Date/Time  Code 

Key  Identification  Code 

Screen  and  Field  Identification  Code 


TASK  13 

(A  Si  B)  (TableNames  T13A  &  T13B) 

Column  Headinc 

Description 

User 

User  Identification  Code 

Date/Time 

Date/Time  Code 

Key 

Key  Identification  Code 

Scr/Fld 

Screen  and  Field  Identification  Code 

TASK  14 

(A  «  B)  (TableNames  T14A  t  T14B) 

Coltimn  Headina 

Description 

User 

User  Identification  Code 

Date/Time 

Date/Time  Code 

Key 

Key  Identification  Code 

Scr/Fld 

Screen  and  Field  Identification  Code 

TASK  15 

(A  6  B)  (TableNames  T15A  fc  TlSB) 

Column  Headina 

Description 

User 

User  Identification  Code 

Date/Time 

Date/Time  Code 

Key 

Key  Identification  Code 

Scr/Fld 

Screen  and  Field  Identification  Code 

LIN 

LIN  Number 
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INSERTION  CRITERIA  TO  TASK  HISTORY  TABLES 
TASK  11  (A  t  B) 

Table  Inclusion  Logic;  Entry  is  made  whenever: 

1.  The  Key  ID  matches  one  of  those  specified  in  the  Key  Collection 
IDs  Section  of  this  task  (See  below) . 

OR 

2.  The  Screen/Field  ID  changes  its  value.  While  in  a  screen  where 
a  LIN  column  exists,  if  the  value  under  LIN  column  matches  one  of 
those  specified  in  the  other  Collection  Attributes  Section  below, 
the  value  of  LIN  will  be  inserted  (otherwise  insertions  will  have 
NULL  LIN  values) . 


Key  Collection  IDs;  Entries  will  be  made  into  the  THT  any  time  any 
of  the  following  keys  or  keytypes  are  pressed: 

<CK> 

Arrow  Keys 

All  Integer  Keys 

<->  (the  minus  sign) 

<ESC> 

<DEL> 

FI 

F2 

Non-Dynamic  Help 
Dynamic  Help 
<Backspace> 


Other  Collection  Attributes; 


TASK  llA 

LIN 

D01857 

K20163 

T36211 


TASK  IIB 
LIN 

C07440 

C08256 

K20163 
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ENCL  2  cont 


TASK  12 

(A  ( 

B) 

Table  Inclusion 

Locic : 

Entry  is 

made 

whenever: 

1.  The  Key  ID 

matches 

one  of 

those 

specified  within  the  Key 

Collection  IDs  Section  of  this  task  (See  below) . 

OR 

2.  The  Screen/Field  ID  changes  its  value. 


Key  Collection  IDs;  Entries  will  be  made  into  the  THT  any  tiroe  any 
of  tne  following  keys  or  keytypes  are  pressed: 

<CR> 

Right  Arrow  Key 
Left  Arrow  Key 
Up  Arrow  Key 
Down  Arrow  Key 
All  Integer  Keys 
<ESC> 

<DEL> 

Non-Dynamic  Help 
Dynamic  Help 
<Backspace> 


Other  Collection  Attributes;  Not  applicable. 


2 
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TASK  13 


(A  i  B) 


Table  Inclusion  Logic;  Entry  is  made  wnenever: 

1.  The  Key  ID  matches  one  of  those  specified  within  the  Ke 
Collection  IDs  Section  of  this  task.  (See  below) . 

OR 

2.  The  Screen/Field  ID  changes  its  value. 


Key  Collection  IDs;  Entries  will  be  made' into  the  TKT  any  time  an 
of  the  following  keys  or  keytypes  are  pressed: 

<CR> 

Right  Arrow  Key 
Left  Arrow  Key 
Up  Arrow  Key 
Down  Arrow  Key 
<ESC> 

<DEL> 

Non-Dynamic  Help 
Dynamic  Help 
<Backspace> 

All  Integer  Keys 
'X' 

Other  Collection  Attributes;  Not  applicable. 
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KNCT,  2  cont 


TASK  14  (A  t  B) 


Table  Inclusion  Logic:  Entry  is  made  whenever: 

1.  The  Key  ID  matches  one  of  those  specified  within  the  Key 
Collection  IDs  Section  of  this  task  (See  below) . 

OR 

2.  The  Screen/Field  ID  changes  its  value. 


Key  ColleetioD  IDs;  Entries  will  be  made  into  the  THT  any  time  any 
of  the  following  keys  or  keytypes  are  pressed; 

<CR> 

Right  Arrow  Key 
Left  Arrow  Key 
Up  Arrow  Key 
Down  Arrow  Key 
<ESC> 

<DEL> 

Non-Dynamic  Help 
Dynamic  Help 


other  Collection  Attributes;  Not  applicable. 
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TASK  15  (A  fc  B) 


Table  Inclusion  Loaic;  Entry  is  made  whenever: 

1.  The  Key  ID  matches  one  of  those  specified  within  the  Key 
Collection  IDs  Section  of  this  task  (See  below) . 

OR 

2.  The  Screen/Field  ID  changes  its  value.  While  in  a  screen  where 
a  LIN  column  exists,  if  the  value  under  LIN  column  matches  one  of 
those  specified  in  the  Other  Collection  Attributes  Section  below, 
the  value  of  LIN  will  be  inserted  (otherwise  insertions  will  have 
NULL  LIN  values) . 


Key  Collection  IDs;  Entries  will  be  made  into  the  THT  any  time  any 
of  the  following  keys  or  keytypes  are  pressed;  » 

<CR> 

Right  Arrow  Key 
Left  Arrow  Key 
Up  Arrow  Key 
Down  Arrow  Key 
<ESC> 

<DEL> 

Non-Dynamic  Help 
Dynamic  Help 
All  Integer  Keys 
'X' 


Other  Collection  Attributes; 


TASK  15A 

LIN 

C07440 

F54817 

N39848 


TASK  15B 
LIN 

C07440 

N39B48 

U06645 
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HEADER/CLOSPRE  DESCRIPTIOK 
HEADER 

The  Header  will  be  called  at  the  beginning  of  each  session 
while  the  user  is  at  the  CIF  Main  Menu.  Header  entries  will  be 
entered  by  the  Session  Administrator  at  the  start  of  a  session. 
The  following  is  the  format  of  the  Header  Stamp  as  it  should  appear 
in  the  Session  Closure  Report; 

TASK  ID *  * 

USER  ID  * 

HELP?  * 

SESSION  START  ** 

The  format  of  screen  support  of  Header  entries  should  be 
similar.  Entries  should  end  with  <CR>  and  the  <CR>  for  the  "HELP" 
entry  should  trigger  the  recording  of  session  start  time  (which 
need  not  be  echoed  to  the  user) . 


CLOSPRE 

The  Closure  will  be  called  at  the  end  of  a  session  when  the 
user  has  returned  to  the  CIF  Main  Menu.  Closure  entries  will  be 
entered  by  the  Session  Administrator  at  the  end  of  a  session.  Once 
the  Closure  entries  are  complete  then  the  database  will  be 
reinitialized  prior  to  the  start  of  a  new  session.  The  following 
is  the  format  of  the  Closure  Stamp  as  it  should  appear  in  the 
Session  Closure  Report: 

SESSION  STOP  ** _  (Also  echo  previously  recorded  SESSION 

START  to  the  screen) 

REMARKS; 

r  DEAD  END  POINTS  * 

OTHER  _ : _ 


OTHER  is  text  entered  by  the  Session  Administrator.  A  single 
<CR>  is  a  line  delimiter  and  causes  a  linefeed  and  carriage  return. 
Two  successive  <CR>'s  cause  closure  of  the  REMARKS  entry,  which 
causes  assembly  (filename  generation  and  saving)  of  the  Session 
Closure  Report. 


*  -  To  be  entered  by  the  Session  Administrator. 

**  -  To  be  recorded  by  the  computer. 
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ENCLOSURE  4 

QUERY  RESULTS  AND  "nTHER  ITEMS" 
TASK  llA 


miFRY  RESULTS 

QUERY  FORMAT  IN  SCR 

1.  Snprial  Issue  {Manual  Turn  In}  SpecTa'i  Issue 


select  date/time  from  THTllA 

where  user= _  &  key=<CR>  6 

scr/fld=ManTI/DutyMOS. 

select  date/time  from  THTllA 

where  user* _  &  kev=<CR>  i 

scr/fld=ManTI/SISSUE. 

count  *  from  THTllA  where 

user* _  &  key=NDH  & 

scr/fld*ManTI/SISSUE. 

count  *  from  THTllA  where 
user*  &  key®OynHlp  & 

scr/fTd^ManTI/SISSUE. 


Start  of  SISSUE 


End  of  SISSUE 


=  times  NonDynHip 


*  times  DynHlp 


2.  SIZE  {Manual  Turnln-Itms  Rtned}  SIZE 


A.  DO  1857  D01857 

select  date/time  from  THTllA  End  of  SIZE/IQTY  Fid 

where  user* _  &  key=<CR>  &  _ 

scr/fld*ManTI-ItmRtn/IOTY  &  _ 

LIN=D01857.  _ 

count  *  from  THTllA  wnere  r  times  NonDynHIo _ 

user*_^  key*N0H  &  scr/fld* 

ManTI^TtmRtn/SIZE  I  LIN* 

D01857. 

count  *  from  THTllA  wnere  -  times  DynHlp _ 

user*^ _  key«OynHlo  & 

scr/fTd*Man7I-ItmRtn/SIZE 
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to 


&  LIN=D01857. 


select  date/time  from  THTllA  End  of  SIZE/IQTY  Fid 

where  user= _  &  key=<CR>  &  _ 

scr/fld=ManTI-ItmRtn/IOTY  &  _ 

LIN=K20163.  _ 


count  *  ■from  THTllA  where  #  times  NonDvnHlp 

user= _  key=NDH  &  scr/fla= 

ManTI-ltmRtn/SIZE  6  LIN= 

K20163. 

count  *  from  THTllA  where  #  times  DynHlp _ 

user= _  key=DynHlD  & 

scr/f ld=ManTI-ItmRtn/SIZE 
&  LIN=K20163. 


C.  T36211  T36211 

select  date/time  from  THTllA  End  of  SIZE/IQTY  Fid 

where  user* _  &  key=<CR>  &  _ 

scr/fld»ManTI-ItmRtn/IQTY  &  _ 

LIN=T36211.  _ 

count  *  from  THTllA  where  #  times  NonDynHlp _ 

user* _  key=N0H  &  scr/fld= 

ManTI-ltmRtn/SIZE  L  LIN= 

T36211. 

count  *  from  THTllA  where  #  times  DynHlp _ 

user=^ _  key=OynHlD  & 

scr/fTd»Mantl-'ltmRtn/SIZE 
&  LIN=T36211. 


.  Fl/F2-f Manual  TurnIn-Itms  Rtned}  F1/F2 

count  *  from  THTllA  where  user* _  #  times  FI 

&  key'Fl  &  scr/fld=ManTI-ItmRtn/*. 
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count  *  from  THTllA  where  user=  #  times  F2 _ 

A  .y=F2  &  scr/fld=ManTI-ItmRtn/’ . 

count  *  from  THTllA  where  user= _  #  times  NonDynHlp  pressed 

&  key=NDH  &  scr/fld=ManTI-ltniRtn7*.  in  ManTI  scr _ 

count  *  from  THTllA  where  user= _  #  times  DynHIp  pressed 

&  key=DynHlp  &  scr/fld=  in  ManTI  scr 

ManTI-ItmRtn/*. 


4.  S$N  {Turn  In  Screen}  SSN 

select  date/time  from  THTllA  where  End  of  SSN 
user=  &  key=<CR>  &  scr/f1d= 

Turnln/SSN. 


count  *  from  THTllA  where  user® _  i?  integers  pressed 

&  key  in(any  integer)  & 
scr/fld=TurnIn/SSN. 

count  *  from  THTllA  where  user* _  #  dashes  pressed 

&  key='-'  &  scr/fld*TurnIn/SSN. 

count  *  from  THTllA  where  user* _  #  times  NonDynHlp 

&  key*NDH  &  scr/fld=TurnIn/SSN. 

count  *  from  THTllA  where  user*  #  times  DynHIp 
&  key*DynHlp  &  scr/fld=TurnIn/SSN. 


5.  ilK.  {Manual  Turnin  Screen)  i)l£ 

select  date/time  fromTHTllA  where  Start  of  UIC _ 

user*  &  (key=<CR»  OR  key*<DN>) 

S.  scr77Td*ManTI/DER0S. 

select  date/time  fromTHTllA  where  End  of  UK 
user*  &  (key*<CR>  OR  key*<0N>) 

&  scr77Td-ManTI/UIC. 

count  *  from  THTllA  where  user* _  #  times  NonDynHlp 

&  key»NDH  &  scr/fld=ManTI/UIC. 
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count  *  from  THTllA  where  user= _ 

&  key=DynHlp  &  scr/f 1 d=ManTI/UIC . 


r  times  DynHlp 


6.  GRADE  {Manual  Turnln  Screen}  GRADE 

select  date/time  fromTHTllA  where  Start  of  GRADE 

user=_^ _  &  (key=<CR>  OR  key=<DN>) 

&  scr7fTd=ManTI/UIC. 


select  date/time  fromTHTllA  where  End  of  GRADE _ 

user=  &  (key=<CR>  OR  key=<0N=*) 

&  scr/fld=ManTI /GRADE. 

count  *  from  THTllA  where  user= _  #  times  NonDynHlp 

&  key=NDH  &  scr/fld=ManTI/GRADE. 

count  *  from  THTllA  where  user= _  i?  times  DynHlp _ 

&  key=DynHlp  &  scr/fld=ManTI/GRADE. 


7.  ,TRAJSFIBRED..1TEM?  TRANSFERRED  ITEMS 

{Manual  Turnln  -  Items  Transferred) 

select  date/time  from  THTllA  where  End  of  Screen _ 

user= _  &  key=<ESC>  &  scr/fld= 

ManTI-ItmTrn/*. 

count  *  from  THTllA  where  user=  =  <CR>s _ 

&  key=<CR>  &  scr/fld*ManTI-ItmTrn/*. 


count  *  from  THTllA  where  user= _  #  Arrow  Keys _ 

&  (key=<UP>  or  key=<DN>  or  key=<LT> 
or  key=<RT>)  &  scr/fld=ManTI-ItmTrn/*. 

count  *  from  THTllA  where  user=  #  times  NonDynHlp 
&  key=NDH  &  scr/fld=ManTI-ItmTrn7*. 

count  *  from  THTllA  where  user* _  #  times  DynHlp _ 

&  key»DynHlp  &  scr/fld=ManTIItmTrn/*. 
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Total  #  keys  pressed  while  in  #keys  prsd(SISSUE) _ 

Special  Issue  field  of  the 
Manual  Turn  In  screen. 

2.  SIZE  {Manual  Turnin-Itms  Rtn}  iUE 

Date/time  of  entry  into  SIZE  fid  Enter  D01857 _ _ 

for  LIN  001857. 

Date/time  of  entry  into  SIZE  fid  Enter  K20163 _ 

for  LIN  K20163. 

Date/time  of  entry  into  SIZE  fid  Enter  T36211 - 

for  LIN  T36211. 

3.  iSH  {Turn  Ins  screen}  iSH 

Date/time  of  entry  into  Turnlns  Enter  SSN _ 

Screen. 


4.  ill£  {Manual  Turnin  screen}  Ul£ 

Total  #  keys  pressed  while  in  #  keys  pre5sed(UIC) - - 

UK  field. 


5.  GRADE  {Manual  Turnin  screen}  SEME 

Total  #  keys  pressed  while  in  #  keys  prsd(GRADE) - 

GRADE  field. 


{Manual  Turn  In  -  Items  Transferred  Screen} 
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Date/time  of  entry  into  screen.  Start  of  Trnsfd  Itms 
Total  #  keys  pressed  in  screen.  #  keys  prsd(Trnfltm) 


Record  the  actual  entries  in  the 
screen  at  the  time  <ESC^  is 
pressed. 


TASK  IIB 


select  date/time  from  THTllB  Start  of  SISSUE 

where  user* _  &  key*<CR>  & 

scr/fld*ManTI/DutyMOS. 


select  date/time  from  THTllB  End  of  SISSUE _ 

where  user* _  &  key*<CR>  & 

scr/fld*ManTI/SISSUE. 

count  *  from  THTllB  where  #  times  NonDynHlp 

user* _  &  key*N0H  & 

scr/fld=ManTI/SISSUE. 

count  *  from  THTllB  where  #  times  DynHlp _ 

user* _  &  key=DynHlp  & 

scr/fld*ManTI/SISSUE. 


2.  SIZEfManual  Turnln-Itms  Rtned}  SIZE 
A.  CQ744Q  C07440 

select  date/time  from  THTllB  End  of  SIZE/IQTY  Fid 
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where  user= _  &  key=<CR>  & 

•.cr/fld=ManTI-ItmRtn/IQTY  & 
LIN=C07440. 


count  *  from  THTllB  where  #  times  NonDynHlp _ 

user= key=NDH  &  scr/f1d= 

ManTI-ItmRtn/SIZE  &  LIN= 

C07440. 

count  *  from  THTllB  where  ?  times  DynHlp _ 

user= key=OynHlp  & 

scr/f Id'ManTI-ItmRtn/SlZE 
&  LIN=C07440. 

B.  K20163  K20163 

select  date/time  from  THTllB  End  of  SIZE/IQTY  Fid 

where  user= _  &  key=<CR>  &  _ 

scr/f1d=ManTI-ItmRtn/I0TY  &  _ 

LIN=K20163.  _ 

count  *  from  THTllB  where  #  times  NonDynHlp _ 

users _  key=NDH  &  scr/fld* 

ManTI-ItmRtn/SIZE  &  LIN= 

K20163. 

count  *  from  THTllB  where  #  times  DynHlp _ 

user= _  key=DynHlp  & 

scr/fla=ManTI-ItmRtn/SIZE 
&  LIN=K20163. 


C.  C08256  mZhh 

select  date/time  from  THTllB  End  of  SIZE/IQTY  Fid 
where  user= _  &  key=<CR>  & 

scr/fld=ManTI-itmRtn/IQTY  &  _ 

LINsC08256.  _ 

count  *  from  THTllB  where  #  times  NonDynHlp _ 

user* _  key=NDH  &  scr/f1d= 

ManTI-ItmRtn/SIZE  &  LIN= 

C08256. 
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count  *  from  THTllB  where  f  times  DynHlp 

user= _  key=OynHlp  & 

scr/f 1o=ManTI -ItmRtn/SIZE 
&  LIN=C08256. 


3.  Fl/F2-f Manual  Turnin-Itms  Rtned}  F1/F2 


count  *  from  THTllB  wnere  user=^ _  *  times  FI _ 

&  key=Fl  &  scr/f ld=ManTI-ltmRtn/’ . 

count  *  from  THTllB  where  user=^ _  #  times  F2 _ 

&  key=F2  &  scr/fld=ManTI-ItmRtn7v 

count  *  from  THTllB  where  user= _  #  times  NonDynHlp  pressed 

&  key=NDH  &  scr/f1d=ManTI-ltmRtn/’'.  in  ManTI  scr _ 

count  *  from  THTllB  where  user= _  #  times  DynHlp  pressed 

&  key=DynHlp  a  scr/fld=  in  ManTI  scr _ 

ManTI-ItmRtn/* . 


4.  {Turn  In  Screen}  55N 

select  date/time  from  THTllB  where  End  of  SSN _ 

user=  &  key=<CR>  &  scr/f ld= 

TurnInTSSN. 

count  ■*  from  THTllB  where  user* _  #  integers  pressed 

&  key  in(any  integer)  & 
scr/fld“TurnIn/SSN. 

count  *  from  THTllB  where  user= _  #  dashes  pressed _ 

&  key='-'  &  scr/fld=TurnIn/SSN. 

count  *  from  THTllB  where  user= _  #  times  NonDynHlp_ 

&  key=NDH  &  scr/fld=TurnIn/SSN. 

count  *  from  THTllB  where  user* _  #  times  DynHlp _ 

&  key=DynHlp  &  scr/fld*Turnln/SSN. 
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5.  iliC  {Manual  Turn  In}  UK 

select  date/time  fromTHTllB  where  Start  of  UIC _ 

user= _  &  (key=<CR>  OR  key=<DN>) 

&  5cr7TTd=ManTI/DER0S. 

select  date/time  fromTHTllB  where  End  of  UIC _ 

user= _  i  (key=<CR=‘  OR  key=<DN>) 

&  scrTfTd-ManTI/UIC. 

count  *  from  THTllB  where  user= _  #  times  NonOynHlp 

&  key=NDH  &  scr/f1d=ManTl/UIC. 

count  *  from  THTllB  wnere  user= _  *  times  DynHlp _ 

&  key=DynHlp  &  scr/fld=ManTI/UIC. 

6.  GRADE  {Manual  Turn  In}  GRADE 

select  date/time  fromTHTllB  where  Start  of  GRADE _ 

user*  &  (key=<CR>  OR  key=<DN>) 

&  scr77Td»ManTI/UIC. 

select  date/time  fromTHTllB  where  End  of  GRADE _ 

user= _  &  {key*<CR>  OR  key=<0N>) 

&  scr77Td*ManTI /GRADE. 

count  *  from  THTllB  where  user= _  #  times  NonDynHlp 

&  key=NDH  &  scr/fld=ManTI /GRADE. 

count  *  from  THTllB  where  user= _  #  times  DynHlp _ 

&  key=OynHlp  &  scr/fld=ManTI/6RADE. 

7.  TRANSFERRED  ITEMS  TRANSFERRED  ITEMS 

{Manual  Turn  In  -  Items  Transferred} 

select  date/time  from  THTllB  where  End  of  Screen _ 

user= _  &  key*<ESC>  &  scr/fld= 

ManTI-ItmTrn/*. 

count  *  from  THTllB  where  user= _  #  <CR>s _ 

&  key*<CR»  &  scr/f 1d=ManTI-ItmTfn7^ . 

count  *  from  THTllB  where  user= _  i  Arrow  Keys _ 

&  (key=<UP>  or  key=<0N>  or  key=‘'LT> 
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or  key=<RT>)  &  scr/fl(j=ManTI-ItmTrn/*. 

count  *  from  THTllB  where  user=  times  NonDynHIp 

&  key=N0H  &  scr/fld=ManTI-ItmTrrr7*. 

count  *  from  THTllB  where  user= _  *  times  DynHlo _ 

&  key=DynHlp  &  scr/fld=ManTIItmTrn/’' . 


"",Tr!FH  ITEMS"  TO  BE  COLLECTED  {IIB} 

DESCRIPTION  FORMAT  IN  SCR 

l.Soecial  Issue  {Manual  Turnin}  Special  Issue 

Total  ?  keys  pressed  while  in  #Keys  prsd(SISSLIE) 

Special  Issue  field  of  the 
Manual  Turn  In  screen. 


2.  SIZE  {Manual  Turnln-Itms  Rtn}  SIZE 


Date/time  of  entry  into  SIZE  fid 
for  LIN  C07440. 

Enter  C07440 

Date/time  of  entry  into  SIZE  fid 
for  LIN  K20163. 

Enter  K20163 

Date/time  of  entry  into  SIZE  fid 
for  LIN  C08256. 

Enter  C08256 

3.  {Turn  Ins  screen} 

SSN 

Date/time  of  entry  into  Turnlns 
Screen. 

Enter  SSN 

4.  UI£  {Manual  Turnin  screen} 

UIC 

Total  F  keys  pressed  while  in 

UIC  field. 

=  kevs  Dressed(UlC) 

5.  GRADE  {Manual  Turnin  screen} 

QRADE 

Total  #  keys  pressed  while  in 
GRADE  field. 

#  kevs  Drsd(GRADE) 
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6.  TPflN«;FFRRFD  ITEMS  JRAN^jF^RRED  IT£^ 

{Manual  Turn  In  -  Items  Transferred  Screen) 

Date/time  of  entry  into  screen.  Start  of  Trnsfd  Urns 

Total  r  keys  pressed  in  screen.  #  keys  prsd(Trnfltm) 


Record  the  actual  entries  in  the 
screen  at  the  time  <ESC>  is 
pressed. 


TRAN<;FFRREn  ITEM  POSTINSS 

UN  iiii  on 


TASK  12A 


nilERY  RESULTS 
QUERY 

1 . 1  r.nndition  Code  E$C-K^ 
{lurn  In  to  DRMO  Screen) 

select  date/time  from  THT12A 

where  user* _  &  key*<ESC>  & 

(scr/fld*TItoDRMO/*  & 

scr/fld<>TItoDRMO/Condition) . 


FHRMAT  IN  SCR 

r.nnflition  Code  &  ESC  Key 

Enter  Condition _ 


select  date/time  from  THT12A  Exit  Condition - 

where  user* _  &  (key*<CR>  or 

key=<ESC>)  &  scr/fld=TItoDRMO/ 

Condition. 

count  *  from  THT12A  where  user* _  #  times  NonDynHlp 

&  key=N0H  &  scr/fld=TItoDRMO/ 

Condition. 


count  *  from  THT12A  where  user*  #  times  DynHlo - 

&  key*OynHlp  &  scr/fld=TItoDRM07 
Condition. 

2.1  rnrrprtions  &  ESC  Key  Corrernons  ^  ESC  t<iy 

{S/C  Screen} 

select  date/time  from  THT12A  Exit  S/C  Screen - - 
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where  user= _  &  (key=<CR>  or 

key=<ESC>)  &  scr/f 1 d=S/C/TotPri  . 

count  *  from  THT12A  where  user= _  it  UP  Arrow _ 

&  key=<UP>  &  scr/f1cl=S/C/* 

count  *  from  THT12A  where  user= _  #  Down  Arrow _ 

&  key=<DN>  &  scr/fl cl=S/C/* 

count  *  from  THT12A  where  user= _  #  Right  Arrow _ 

&  key=<RT>  &  scr/f 1 d=S/C/* 

count  *  from  THT12A  where  user= _  #  Left  Arrow _ 

&  key=<LT>  &  scr/f1d=S/C/* 

count  *  from  THT12A  where  user= _  #  Backspaces _ 

L  key=<BCKSP>  &  scr/fld=S/C/* 

count  *  from  THT12A  where  user= _  #  times  NonDynHIp 

&  key=NDH  &  scr/f 1 d=S/C/* . 

count  *  from  THT12A  where  user= _  #  times  DynHlp _ 

&  key=DynHlp  &  scr/f1d=S/C/*. 

count  *  from  THT12A  where  user= _  i  integers _ 

&  key  in (any  integer)  & 
scr/fld=S/C/*. 

3 . 1  Menu  Selection  Menu  Selection 
{Adjustment  Transactions  Menu  to  Cancel  S/C,R/S,C/C} 

count  *  from  THT12A  where  user= _  #  times  NonDynHIp 

&  key=NDH  &  date/time  from 
&entadjtrns  to  &entcancrepsur. 

count  *  from  THT12A  where  user= _  ?  times  DynHlp _ 

&  key=DynHlp  &  date/time  from 
&entadjtrns  to  &entcancrepsur. 

3.2  Total  Price  Total  P-ice 

{Cancel  S/C,R/S,C/C} 

select  date/time  from  THT12A  Enter  Total  Price_ 

where  user= _  &  key=<ESC>  & 
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(scr/f''^=Cancelsyc.R/S/*  & 
scr/i  ld<=“CancelS/C,R/S/TotPri ) . 

select  date/time  from  THT12A  Exit  Total  Price _ 

where  user= _  &  (key=<CR:  or 

key=<ESC>)  &  scr/fld=Cance1S/C, 

R/S/TotPri. 

count  *  from  THT12A  where  user= _ _  #  times  NonDynHlp 

&  key=NDH  &  scr/fld=CancelS/C, 

R/S/TotPri . 

count  *  from  THT12A  where  user=  #  times  DynHlp _ 

&  key=DynHlp  &  scr/fld=CancelS/C, 

R/S/TotPri . 


"Qjm. -nEMSI’-TQ.  6,E..CQUt.E.CI5P  {12A} 


DESCRIPTION  FORMAT  IN  SCR 

1.  Condition  Code  &  ESC  Kev  Condition  Code  &  ESC  Kev 

{Turnin  to  DRMO  Screen) 

Record  date/time  of  entry  Enter  TltoDRMO _ 

into  the  Turnin  to  DRMO  Screen. 


Total  #  keys  pressed  in  the  Total  #  key5(TlDRM0) 

Turnin  to  DRMO  Screen. 


Total  #  keys  pressed  in  the  Total  #  keys(Condition) 

Condition  Code  field  of  the 
Turnin  to  DRMO  Screen. 

E-SROR-CimS 

Record  the  NSNs  &  quantities  on  i/C 

the  screen  at  the  time  <ESC>  key  HSN  Oil 

is  pressed  and  scr/fld<»TItoDRMO/  _  _ 

Condi tion. 


Record  the  Condition  Cooe  on  the  Condition  Code 
Screen  at  time  of  exit  from  screen. 
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2.  Corrections  &  ESC  Kev 
{S/C  Screen} 

Record  the  date/time  of  entry 
into  the  Statement  of  Charges 
Screen  from  the  Adjustment 
Transactions  Menu  Screen. 

Total  #  keys  pressed  in  the 
S/C  Screen. 

Total  #  keys  pressed  in  the 
Total  Price  field  of  the 
S/C  Screen. 


Corrections  &  ESC  Kev 
Enter  S/C  Screen 


Total  #  keys(S/C) _ 

Total  i  keys(TotPri) 


ERROR  CHECKS 

Record  the  NSNs  &  quantities  on  i/C 

the  screen  at  the  time  <ESC>  key  NiN  QTY 

is  pressed  and  scr/fld<>S/C/TotPri .  _  _ 

Record  the  Total  Price  on  the  Total  Price _ 

Screen  at  time  of  exit  from 
screen. {key=<CR>  or  <ESC>} 


3.1  Menu  Selection  Menu  Selection 

{Adjustment  Transactions  Menu  to  Cancel  S/C,R/S,C/C} 

{&entadjtrns}  =  the  date/time  Enter  Adj  Tran5_ 

when  user  reentered  the  Adj 
Trans  Menu  Screen  from  previous 
subtask  {#12A.2}. 

{&entcancrepsur}  =  the  date/time  Enter  Cancel  R/S 
of  entry  into  Cancel  S/C, R/S, C/C 
Screen  from  the  Adj  Trans  Menu 
Screen. 


Record  total  #  key  presses  made  *  keys  pressed  to  get 

from  time  of  reaching  Adj  Trans  to  Cancel  R/S  _ 

Menu  Screen  after  Subtask  12A.2 
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to  time  of  entering  the  Cancel 
S/C,R/S,C/C  Screen. 


3.2  Total  Price 

{Cancel  S/C,R/S,C/C  Screen} 

Total  #  keys  pressed  m  Total 
Price  field  of  the  Cancel  S/C, 
R/S.C/C  Screen. 

ERROR  CHECKS 


Total  Price 

#  keys  in  Total  Price 
of  Cancel  R/S 


Record  the  NSNs  &  quantities  on  Cancel  R/S 

the  screen  at  the  time  <ESC>  key  NSN  OTY 

is  pressed  and  scr/fld<>Cancel  _  _ 

S/C,R/S,C/C/TotPri .{ie  NSNs  &  Qtys 
at  time  of  entry  into  Tot  Pri} 


Record  the  Total  Price  on  the  Total  Cancelled 

Screen  at  time  of  exit  from 
Cancel  S/C, R/S, C/C  Screen. 

{key=<CR>  or  <ESC>} 


TASK  12B 


QUERY  RESULTS 
aUERY 

1.1  Condition  Code  &  ESC  Kev 
{Turn  In  to  DRMO  Screen} 

select  date/time  from  THT12B 

where  user= _  &  key=<ESC>  & 

(scr/fld=TItoORMO/»  & 
scr/fld<»TItoDRMO/Condition) . 

select  date/time  from  THT12B 


FORMAT  IN  SCR 

Condition  Code  £  ESC  Kev 

Enter  Condition _ 

Exit  Condition _ 
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where  user= _  &  (|(ey=<CR>  or 

key=<ESC>)  &  scr/fld=TItoDRMO/ 

Condition. 

count  *  from  THT12B  where  user= _  #  times  NonDynHlp 

&  key=NDH  &  scr/fld=TItoDRMO/ 

Condition . 

count  *  from  THT12B  where  user=^ _  #  times  DynHlp _ 

&  key=DynHlp  &  scr/f 1 d=TItoDRM07 
Condition . 


2. 1  Corrections  &  ESC  Kev  Corrections  &  ESC  Kev 

{S/C  Screen} 


select  date/time  from  THT12B  Exit  S/C  Screen 

where  user= _  &  (key=<CR>  or 

key=<ESC>)  &  scr/f 1 d=S/C/TotPri . 

count  *  from  THT12B  where  user=  #  UP  Arrow 
&  key=<UP>  &  scr/fld=S/C/* 

count  ■*  from  THT12B  where  user= _  #  Down  Arrow _ 

&  key=<DN>  &  scr/f 1 d=S/C/» 

count  *  from  THT12B  where  user= _  #  Right  Arrow_ 

&  key=<RT>  &  scr/f 1 d=S/C/* 


count  *  from  THT12B  wnere  user= _  #  Left  Arrow 

&  key=<LT>  &  scr/fld=S/C/'* 


count  *  from  THT12B  where  user=  #  Backspaces 
&  key=<BCKSP>  &  scr/fld=S/C/* 

count  *  from  THT12B  where  user= _  #  times  NonDynHlp 

&  key=NDH  &  scr/fld=S/C/*. 

count  *  from  THT12B  where  user= _  #  times  DynHlp _ 

&  key=DynHlp  &  scr/fl d=S/C/*. 


count  *  from  THT12B  where  user= _  #  integers 
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&  key  in(any  integer)  & 
scr, 1 1 d=S/C/* . 


3. 1  Menu  Selection  Menu  Selection 
{Adjustment  Transactions  Menu  to  Cancel  S/C,R/S,C/C} 

count  *  from  THT12B  where  user= _  #  times  NonDynHlp 

&  key=N0H  &  date/time  from 
&entadjtrns  to  &entcancrepsur . 

count  *  from  THT12B  where  user= _  #  times  DynHlp _ 

&  key=0ynH1p  &  date/time  from 
feentadjtrns  to  &entcancrepsur . 

3.2  Total  Price  Total  Price 

{Cancel  S/C,R/S,C/C} 

select  date/time  from  THT12B  Enter  Total  Price_ 

where  user= _  &  key=<ESC>  & 

(scr/fld=CancelS/C,R/S/*  & 
scr/fld<^CancelS/C,R/S/TotPri ) . 


select  date/time  from  THT12B  Exit  Total  Price _ 

where  user* _  &  (key*<CR>  or 

key=<ESC>)  &  scr/fld=CancelS/C, 

R/S/TotPri . 

count  *  from  THT12B  where  user= _  #  times  NonDynHlp 

&  key=NDH  &  scr/fld=CancelS/C, 

R/S/TotPri . 

count  *  from  THT12B  where  user* _  #  times  DynHlp _ 

&  key=OynHlp  &  scr/fld=CancelS/C, 

R/S/TotPri . 
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Record  date/time  of  entry  Enter  TItoDRMO 

into  the  Turnln  to  DRMO  Screen. 


Total  #  keys  pressed  in  the  Total  #  keys(TIDRMO) 

Turnln  to  DRMO  Screen. 


Total  #  keys  pressed  in  the  Total  #  keys(Condi tion) 

Condition  Code  field  of  the 
Turnln  to  DRMO  Screen. 

ERROR  CHECKS 

Record  the  NSNs  &  quantities  on  S/C 

the  screen  at  the  time  <ESC>  key  N^N  QIY 

is  pressed  and  scr/fld<>TItoDRMO/  _  _ 

Condition.  _  _ 

Record  the  Condition  Code  on  the  Condition  Code _ 

Screen  at  time  of  exit  from  screen. 


2.  Corrections  &  ESC  Kev 
{S/C  Screen} 

Record  the  date/time  of  entry 
into  the  Statement  of  Charges 
Screen  from  the  Adjustment 
Transactions  Menu  Screen. 

Total  #  keys  pressed  in  the 
S/C  Screen. 

Total  #  keys  pressed  in  the 
Total  Price  field  of  the 
S/C  Screen. 


Corrections  &  ESC  Kev 


Enter  S/C  Screen 


Total  #  keys(S/C) 


Total  #  keys(TotPri) 


ERROR  CHECKS 

Record  the  NSNs  &  quantities  on  S/C 

the  screen  at  the  time  <ESC>  key  NSN  QTY 

is  pressed  and  scr/fld<>S/C/TotPri .  _  _ 

Record  the  Total  Price  on  the  Total  Price _ 

Screen  at  time  of  exit  from  screen. 
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{kiy=<CR>  or  <ESC>} 


3 . 1  Menu  Selection  Menu  Selection 

{Adjustment  Transactions  Menu  to  Cancel  S/C,R/S,C/C} 


{&entadjtrns}  =  the  date/time  Enter  Adj  Trans_ 
when  user  reentered  the  Adj 
Trans  Menu  Screen  from  previous 
subtask  {#12B.2}. 

{&entcancrepsur}  =  the  date/time  Enter  Cancel  R/S 
of  entry  into  Cancel  S/C.R/S,C/C 
Screen  from  the  Adj  Trans  Menu 
Screen. 


Record  total  #  key  presses  made 
from  time  of  reaching  Adj  Trans 
Menu  Screen  after  Subtask  12B.2 
to  time  of  entering  the  Cancel 
S/C.R/S,C/C  Screen. 

3.2  Total  Price 

{Cancel  S/C, R/S, C/C  Screen} 

Total  #  keys  pressed  in  Total 
Price  field  of  the  Cancel  S/C, 
R/S, C/C  Screen. 

ERROR  CHECKS 


F  keys  pressed  to  get 
to  Cancel  R/S 


Total  Price 

#  keys  in  Total  Price 
of  Cancel  R/S 


Record  the  NSNs  &  quantities  on 
the  screen  at  the  time  <ESC>  key 
is  pressed  and  scr/f1d<>Cancel 
S/C,R/S,C/C/TotPri. 

Record  the  Total  Price  on  the 
Screen  at  time  of  exit  from 
Cancel  S/C, R/S, C/C  Screen. 
{key=<CR>  or  <ESC>} 


Cancel  R/S 
NSN  QTY 


Total  Cancelled 


TASK  13A 


QUERY  RESULTS 
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QUERY  FORMAT  IN  SCR 

1 .  Reouired  Fields  &  Correction 
See  "Other  Items”  to  be  Collected. 


2.  Menu  Selection  Menu  Selection 

{Subtask  13A.3} 

{Document  Register  Actions  Menu  to  Post  Status  Changes  and/or 
Cancellations} 

count  *  from  THT13A  where  user= _  #  Arrow  keys  from  Doc 


&  (key=<UP>  or  key=''DN>  or  key 
or  key=<LT>)  &  date/time  from 
{&entdocreg}  to  {&entDostcanc} 

count  *  from  THT13A  where  user 
&  (key  in (any  integer)  or  key= 
&  date/time  from  {&entdocreg} 
to  {&entpostcanc} . 

count  *  from  THT13A  where  user 
&  key*<CR>  &  date/time  from 
{&entdocreg}  to  {ientoostcanc} 

count  *  from  THT13A  where  user’ 
&  key=NDH  &  date/time  from 
{&entdocreg}  to  {&entpostcanc} 

count  *  from  THT13A  wnere  user 
&  key=DynHlp  &  date/time  from 
{&entdocreg}  to  {&entpostcanc} 


<RT>  Reg  Menu  to  Post  Ch/Canc 
Screen 


_ _  #  integers  or  'X’chosen 

X')  between  Doc  Reg  Menu  to 
Post  Ch/CancScreen 


#  <CR>'s  between 
Doc  Reg  Menu  and 

Post  Ch/CancScreen _ 

#  times  NonDynHlp  pressed 

between  Doc  Reg  Menu  and 
Post  Ch/CancScreen _ 

#  times  DynHlp  pressed 
between  Doc  Reg  Menu  and 
Post  Ch/CancScreen 


"OTHER  ITEMS"  TO  BE  COLLECTED  {13A} 

DESCRIPTION  FORMAT  IN  SCR 

1.  Required  Fields  &  Correction  Required  Fields  &  Correction 
{Subtasks  13A.1  &  13A.2} 

{Create  Supply  Requisitions) 
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For  each  sojourn  in  the  Create 
Supply  Requisitions  Screen, 
collect  the  following; 

Date/time  of  entry 
Date/time  of  exit 

#  <UP>  keys  pressed 
a  <DN>  keys  pressed 

#  <0EL>  keys  pressed 

#  <ESC>  keys  pressed 

#  <CR>'s  pressed 

#  <BCKSP>  keys  pressed 

#  times  NonDynHlp  pressed 
a  times  DynHlp  pressed 

#  keystrokes  per  screen 


Entry  Date/time _ 

Exit  Date/time _ 

UP  keys  _ 

#  Down  keys _ 

#  DEL  keys _ 

#  ESC  keys _ 

#  <CR>'s _ 

#  Backspaces _ 

?  times  NonDynHlp_ 

#  times  DynHlp _ 

Total  #  keystroxes 


ERROR  CHECKS 

Record  the  following  items  at 
the  time  of  each  non-delete  type 
exit  from  the  Create  Supply 
Requisitions  Screen: 

Document  Number  (Date  &  Serial) 
NSN 

Document  Identifier  Code 
Quanti ty 
Demand  Code 
Priority  Code 


2.  Menu  Selection 
{Subtask  13A.3} 

{Document  Register  Actions  Menu  to  Post  Status  Changes  and/or 
Cancellations} 

{&entdocreg}  =  date/time  of  Enter  Doc  Reg  Actions _ 

entry  into  the  Document  Register 
Actions  Menu  from  the  Create  Supply 
Requisitions  Screen  at  the  end  of 
Subtask  13A.2. 

<  -  :i 


REQUISITIONS 

Date  Ser  NSN  PIC  QTY  PHD  PRI 


Menu  Selection 
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{&entpostcanc}  =  date/time  of  Enter  Post  Changes/ 

every  entry  into  the  Post  Status  Cancellations 

Changes  and/or  Cancellations  Screen _ 

Screen  from  the  Doc  Reg  Menu. 


Record  #  keystrokes  when  date/time 
value  is  between  START  and  END  as 
defined  above(Query-Menu  Selection) 


#  keystrokes  between  Doc 
Reg  Actions  Menu  &  Post 
Changes /Cancel  1 ati ons 
Screen 


ERROR  CHECKS 

Record  the  following  items  at  time 
of  exit  from  the  Post  Status  Change 
and/or  Cancellation  Screen: 


Document  #  (Date  &  Serial)  REQUISITIONS 

NSN  DATE  SER  NSN  QTYREQ  STAT  QTYCANC  EDD 

Quantity  Requested  _ 

Status  Code 

Expected  Delivery  Date 
Quantity  Cancelled 


TASK  13B 


QUERY  RESULTS 

QUERY  FORMAT  IN  SCR 

1.  Required  Fields  &  Correction 
See  "Other  Items"  to  be  Collecteo. 

2 .  Menu  Selection  Menu  Selection 
{Subtask  13B.3} 

{Document  Register  Actions  Menu  to  Post  Status  Changes  and/or 
Cancellations} 

count  *  from  THT13B  where  user= _  #  Arrow  keys  from  Doc 

&  (key=«UP>  or  key=<DN>  or  key=<RT^  Reg  Menu  to  Post  Ch/Canc 

or  key=<LT>)  &  date/time  from  Screen _ _ 

{&entdocreg}  to  {&entpostcanc} . 
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count  *  from  THT13B  where  user= 
&  (key  in(any  integer)  or  key= 
&  date/time  from  {&entdocreg} 
to  {&entpostcanc} . 

count  *  from  THT13B  where  user' 
&  key=<CR>  &  date/time  from 
{&entdocreg}  to  {&entpostcanc} 

count  *  from  THT13B  where  user 
&  key=NDH  &  date/time  from 
{&entdocreg}  to  {&entpostcanc} 

count  *  from  THT13B  where  user 
&  key=DynHlp  &  date/time  from 
{&entoocreg}  to  {Aentpostcanc} 


_  integers  or  'X'chosen 

rr  between  Doc  Reg  Menu  to 
Post  Ch/CancScreen _ 


^  <CR>'s  between 
Doc  Reg  Menu  and 
Post  Ch/CancScreen _ 

#  times  NonDynHlp  pressed 

between  Doc  Reg  Menu  anc 
Post  Ch/CancScreen _ 

*  times  DynHlp  pressed 

oetween  Doc  Reg  Menu  anc 
Post  Ch/CancScreen _ 


'■OTHER  ITEMS"  TO  BE  COLLECTED  {13B} 


DESCRIPTION 

1.  Reouired  Fields  &  Correctior 
{Subtasks  13B.1  &  13B.2} 

{Create  Supply  Requisitions} 

For  each  sojourn  in  the  Create 
Supply  Requisitions  Screen, 
collect  the  following: 

Date/time  of  entry 
Date/time  of  exit 

#  <UP>  keys  pressed 
4*  <0N>  keys  pressed 

f?  <DEL>  keys  pressed 

#  <ESC>  keys  pressed 

#  <CR>'s  presseo 

?  <BCKSP>  keys  pressed 

#  times  NonDynHlp  pressed 

#  times  DynHlp  pressed 

#  keystrokes  per  screen 


FORMAT  IN  SCR 

Required  Fields  ^__Correcti on 


Entry  Date/time _ 

Exit  Date/ time _ 

F  UP  keys  _ 

r  Down  keys _ _ 

F  DEL  keys _ 

#  ESC  keys _ 

i  <R>  s _ 

?  Backspaces _ 

r  times  NonDynHlp_ 

i  times  DynHlp _ 

Total  r  keystrokes 
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ERROR  CHECKS 


Record  the  following  items  at 
the  time  of  each  non-delete  type 
exit  from  the  Create  Supply 
Requisitions  Screen; 

Document  Number  (Date  &  Serial)  REQUISITIONS 

NSN  Date  Sea  NSN  QIC  QTY  DMD  PRI 

Document  Identifier  Code  _ 

Quantity  _ 

Demand  Code  _ 

Priority  Code  _ 


2 .  Menu  Selection  Menu  Selection 

{Subtask  136.3} 

{Document  Register  Actions  Menu  to  Post  Status  Changes  and/or 
Cancel lations) 

{&entdocreg}  =  date/time  of  Enter  Doc  Reg  Actions _ 

entry  into  the  Document  Register 
Actions  Menu  from  the  Create  Supply 
Requisitions  Screen  at  the  end  of 
Subtask  13B.2. 


{&entpostcanc}  =  date/time  of  Enter  Post  Changes/ 

every  entry  into  the  Post  Status  Cancellations 

Changes  and/or  Cancellations  Screen__ _ 

Screen  from  the  Doc  Reg  Menu. 

Record  #  keystrokes  when  date/time  #  keystrokes  between  Doc 
value  is  between  START  and  END  as  Reg  Actions  Menu  &  Post 
defined  above(Ouery-Menu  Selection)  Changes/Cancel  1 ations 

Screen  _ 


ERROR  CHECKS 

Record  the  following  items  at  time 
of  exit  from  the  Post  Status  Change 
and/or  Cancellation  Screen: 

Document  ^  (Date  &  Serial)  REQUISITIONS 
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NSN 

Quantity  Requested 
Status  Code 

Expected  Delivery  Date 
Quantity  Cancel  led 


QUERY  RESULTS 

QUERY  FORMAT  IN  SCR 

1 .  Negative  Receipt  &  Reopen  Closed  Documents 
{Post  Receipts  ^  Reopen  Closed  Documents} 

ERROR  CHECKS 

select  docnr_dt,docnr_nm,  FINAL  DOCUMENTS 

qtyreq.qtyrecvd.compdte  from  DATE  SER  OTYREO  OTYRECD  COMP 

aeb09t  where  docnr_dt=0096  _ 

&  (docnr  nm*0048  or  docnr  nm= 

0049).  ■ 


DATE  SER  NSN  QTYREQ  STAT  OTYCANC  EDD 


TASK  14A 


"OTHER  ITEMS”  TO  BE  COLLECTED  {14A} 

DESCRIPTION  FORMAT  IN  SCR 

1 .  Nec  Rcpt  &  Reopen  Closed  Docs  Neo  Rcpt  &  Reopen  Closed  Docs 
{SubtasKS  14A.1  &  14A.2} 

{Post  Receipts  &  Reopen  Closed  Documents) 

For  eacn  sojourn  in  the  Post 
Receipts  Screen, 
collect  the  following; 

Date/time  of  entry 
Date/time  of  exit 

#  times  NonDynHIp  pressed 

#  times  DynHlp  pressed 

#  keystrokes  per  screen 


Post  Receipts 

Enter  Post  Rcpts _ 

Exit  Post  Rcpts 
»  NonDynHIp (PstRcpt) _ 

*  D^nHl p(PostRcpt) _ 

#  keystrokes (PostRcpt ) 
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For  each  sojourn  in  the 
Reopen  Closed  Documents 
Screen,  collect  the  following: 

Date/time  of  entry 
Date/time  of  exit 

#  times  NonDynHlp  pressed 

#  times  DynHlp  pressed 

#  keystrokes  per  screen 


Determine  #  keystrokes  from 
beginning  of  session  to  the 
entry  into  the  Reopen  Closed 
Documents  Screen. 


Post  Receipts 

Enter  REOPEN 
Exit  REOPEN 

#  NonDynHlp(REOPEN)_ 

#  DynHlp(REOPEN) _ 

#  keystrokes ( REOPEN) 


#  keystrokes  from  Start  of 
Session  to  REOPEN 
Screen 


TASK  14B 


QUERY  RESULTS 

QUERY  FORMAT  IN  SCR 

1.  Negative  Receipt  &  Reopen  Closed  Documents 
{Post  Receipts  &  Reopen  Closed  Documents} 

ERROR  CHECKS 

select  docnr_dt,docnr_nm, 
qtyreq,gtyrecvd,compdte  from 
aeb09t  where  docnr_dt=9226 
&  (docnr  nm=0080  or  docnr_nm 
0081). 

"OTHER  ITEMS”  TO  BE  COLLECTED  {lAB} 

DESCRIPTION  FORMAT  IN  SCR 

1.  Neg  Rcpt  &  Reopen  Closed  Docs  Neg  Rcpt  a-  Reopen  Closed  Docs 
{Subtasks  148. 1  &  148.2} 

{Post  Receipts  &  Reopen  Closed  Documents} 

For  each  sojourn  in  the  Post 


FINAL  DOCUMENTS 
DATE  SER  QTYREQ  QTYRECD  COMP 
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Receipts  Screen, 
collect  the  following; 

Date/time  of  entry 
Date/time  of  exit 

#  times  NonDynHlp  pressed 

#  times  DynHlp  pressed 

#  keystrokes  per  screen 

For  each  sojourn  in  the 
Reopen  Closed  Documents 
Screen,  collect  the  following: 

Date/time  of  entry 
Date/time  of  exit 

#  times  NonDynHlp  presseo 

#  times  DynHlp  pressed 

#  keystrokes  per  screen 


Determine  #  keystrokes  from 
beginning  of  session  to  the 
entry  into  the  Reopen  Closed 
Documents  Screen. 


Post  Receipts 

Enter  Post  Rents _ 

Exit  Post  Repts _ 

#  NonDynHlo(PstRcpt) _ 

#  DynHlp(PostRcpt) _ 

i  keystrokes ( PostRcpt) 


Post  Receipts 

Enter  REOPEN 

Exit  REOPEN^ _ 

#  NonDynHl p(RE0PEN)_ 

#  DynHlp(REOPEN) 

#  keystrokes (REOPTnT 


#  keystrokes  from  Start  of 
Session  to  REOPEN 
Screen _ 


TASK  15A 


QUERY  FORMAT  IN  SCR 

1 . 1  Fastest  Movement  &  Correction  Fastest  Movement  &  Correction 
{Complete  the  Issue  Screen} 

select  date/time  from  THT15A  where  Exit  IQTY  of  LIN  F54817 

user= _  &  scr/fld=ComoIssue/IOTY  &  _ 

LIN*F54817  &  (key”<CR>  or  key=<UP>  _ 

or  key*<DN>) . 

select  date/time  from  THT15A  where  Exit  lOTY  of  LIN  C07440 
user® _  &  scr/fld'ComoIssue/IQTY  &  _ 

4-27 


205 


LIN=C07440  &  (key=<CR>  or  key=<UP' 
or  key=<DN>) . 


count  *  from  THT15A  where  user= 
&  key=NDH  &  scr/fld=CompIssue/*' 


count  *  from  THT15A  where  user= _ 

&  key=DynHlp  &  scr/f 1 d=Complssue/’ 


1.2  SIZE  {Complete  the  Issue} 

A.  C07440 

select  date/time  from  THT15A 

where  user= _  &  (key=<CR>  or 

key=<DN>)  &  scr/f 1 d=CompIssue/ 
IQTY  &  LIN=C07440. 

count  *  from  THT15A  where 

user= _  &  key=NDH  &  scr/f ld= 

CompIssue/SIZE  &  LIN=C07440. 

count  *  from  THT15A  where 

user* _  &  key=DynHlp  & 

scr/fld=CompIssue/SIZE  & 
LIN=C07440. 


#  times  NonDynHIp  in 

Complete  the  Issue 
Screen _ 

#  times  DynHlp  in 

Complete  the  Issue 
Screen _ 

iIZ£ 

C07440 

End  of  SIZE/IQTY  Field 


#  times  NonDynHl p(C07440) 


#  times  DynHl p(C07440) 


B.  N39848 

select  date/time  from  THT15A 

where  user= _  &  (key=<CR>  or 

key=<0N>)  &  scr/fld=Complssue/ 
IQTY  &  LIN=N39848. 

count  *  from  THT15A  where 

user* _  &  key*NDH  &  scr/f ld= 

Complisue/SIZE  &  LIN-N39848. 

count  *  from  THT15A  where 

user*^ _  &  key=DynHlD  & 

scr/fld*CompIssue/SIZE  & 


End  of  SIZE/IQTY  Field 


#  times  NonDynHlp(N39848) 


#  times  DynHl p(N39848) 


-  28 


4 


206 


LIN=N''0848. 


2 . 1  Menu  Selection  Menu  Selection 

{Clothing  Issues  Menu  to  Issue  Due-Out  Items] 

select  date/time  from  THT15A  where  EXIT  Clothing  Issues  Menu 

user= _  &  scr/f  1  d=Cl othlssMenu/*  _ _ 

&  (key  in(any  integer)  or  key='X' 

or  key=<CR>).  _ 


select  date/time  from  THT15A  where  Complete  SSN  in  Issue 

user= _  &  scr/f ld=IssueDueOut/SSN  Due-Out  Items  Screen 

&  (key=<CR>  or  key=<RT>).  _ 


2.2  Non-Issue  Quantity  Non-Issue  Quantity 

{Issue  Due-Out  Items} 


select  date/time  from  THTI5A  where 

user= _  &  scr/fld=IssueDueOut/* 

&  key=<ESC>. 

count  *  from  THT15A  where  user=_^ _ 

&  key*=NDH  &  scr/fld=IssueDue0ut7^ 


EXIT  Due-Out  Items 
Screen 


#  times  NonDynHlp  in 
Screen 


count  *  from  THT15A  where  user= _  #  times  DynHlp  in 

&  key=DynHlp  &  scr/fld=  Screen _ 

IssueDueOut/*. 


"OTHER  ITEMS”  TO  BE  COLLECTED  {ISA} 
DESCRIPTION 

1 . 1  Fastest  Movement  &  Correction 
{Complete  the  Issue  Screen} 

Record  date/time  of  entry  into 
the  Complete  the  Issue  Screen. 

Record  #  keystrokes  from  1st 
entry  into  LIN  F54817  and 
re-entry  into  SIZE  fid  of  C07440. 


FORMAT  IN  SCR 

Fastest  Movement  &  Correction 

Entry  into  Complete  the 
Issue  Screen _ 

"  keystrokes  F548I7  to 
C07440  _ 
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EXIT  Complete  the  Issue 
Screen 


SIZE 


Enter  C07440 


Enter  N39848 


Menu  Selection 
Due-Out  Items} 

Re-Enter  Clothing  Issue 
Menu 


#  keystrokes  between 
Clothing  Issue  Menu 
&  Due-Out  Items  Screen 


Record  Date/time  of  exit  from 
screen  (ie  <ESC>  accepted  by 
program) . 

1.2  SIZE  {Complete  the  Issue) 

Date/time  of  entry  into  SIZE 
field  for  LIN  C07440. 


Date/time  of  entry  into  SIZE 
field  for  LIN  N39848. 


2 . 1  Menu  Selection 
{Clothing  Issues  Menu  to  Issue 

Record  date/time  of  re-entry 
into  Clothing  Issue  Process 
Menu  from  the  Complete  the 
Issue  Screen. 


Record  #  keystrokes  between 
re-entry  into  Clothing  Issues 
Process  Menu  and  entry  into 
Issue  Due-Out  Items  Screen. 

Record  #  times  NonOynHlp 
key  pressed  between  re-entry 
into  Clothing  Issues  Process 
Menu  &  Complete  the  Issue  Screen. 

Record  #  times  DynHlp 
key  pressed  between  re-entry 
into  Clothing  Issues  Process 
Menu  &  Complete  the  Issue  Screen. 

2.2  Non-Issue  Quantity 
{Issue  Due-Out  Items) 


#  times  NonDynHlp  between 

Clothing  Issues  Menu  reentry 
&  Issue  Due-Out  Items 
Screen _ 

#  times  DynHlp  between 

Clothing  Issues  Menu  reentry 
&  Issue  Due-Out  Items 
Screen _ 

Non-Issue  Quantity 


Record  r  keystrokes  made  in 


#  keystrokes  in  Issue 


Issue  Due-Out  Items  Screen. 


Due-Out  Items  Screen 


{Complete  the  Issue  Screen} 


select  date/time  from  THT15B  where  Exit  IQTY  of  LIN  U06645 
user=  &  scr/fld=CompIssue/IQTY  & 

LIN=UM645  &  (key=<CR>  or  key=<UP>  _ 

or  key=<DN>) . 

select  date/time  from  THT15B  where  Exit  IQTY  of  LIN  N39848 
user*  &  scr/fld*CompIssue/IQTY  & 

LIN=nM48  &  (key*<CR>  or  key=<UP>  _ 

or  key=<0N>) . 

count  *  from  THT15B  where  user* _  #  times  NonDynHlp  in 

&  key=NDH  &  scr/fl d=CompIssue/* .  Complete  the  Issue 

Screen _ 

count  *  from  THT15B  where  user=__^  #  times  DynHlp  in 
&  key=DynHlp  &  scr/fl d=Complssue7'.  Complete  the  Issue 

Screen _ 

1.2  SIZE  {Complete  the  Issue}  SIZE 

A.  C07440  C07440 

select  date/time  from  THT15B  End  of  SIZE/IQTY  Field 

where  user* _  &  (key=<CR>  or  _ 

key=<DN>)  &  scr/f1d=CompIssue/ 

IQTY  &  LIN*C07440.  _ 

count  *  from  THT15B  where  #  times  NonDynHlp(C07440) 

user*  &  key=NDH  &  scr/flo* 

Complssue/SIZE  &  LIN=C07440. 

4  -  3i 
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count  *  from  THT15B  where 

user=^ _  &  key=DynHlp  & 

scr/f1d=Comp Issue/S  I ZE  & 
LIN=C07440. 

B.  N39848 

select  date/time  from  THT15B 

where  user= _  &  (key=<CR>  or 

key=<DN>)  L  scr/fl d=CompIssue/ 
IQTY  &  LIN=N39848. 

count  *  from  THT15B  where 
user=  &  key=NDH  &  scr/fl d= 

Complssue/SIZE  &  LIN=N39848. 

count  *  from  THT15B  where 

user= &  key=DynHlp  & 

scr/fld=Comolssue/SIZE  & 
LIN=N39848. 


#  times  DynHl p(C07440) 


N39848 

End  of  SIZE/IQTY  Field 


#  times  NonDynHl p(N39848) 


#  times  DynHl p (N39848) 


2.1  Menu  Selection  $elgC..t.?..en 

{Clothing  Issues  Menu  to  Issue  Due-Out  Items} 


select  date/time  from  THT15B  where 

user= _  &  scr/fld=ClothIssMenu/* 

&  (key  in(any  integer)  or  key='X’ 
or  key=<CR>) . 

select  date/time  from  THT15B  where 

user= _  &  scr/fld=IssueOueOut/SSN 

&  (key=<CR>  or  key=<RT>). 

2.2  Non-Issue  Quantity 
{Issue  Due-Out  Items} 

select  date/time  from  THT15B  where 

user= _  &  scr/fld=IssueDueOut/* 

&  key=<ESC>. 

count  *  from  THT15B  where  user= _ 

&  key*=NDH  &  scr/fld=IssueDueOut/*. 


EXIT  Clothing  Issues  Menu 


Complete  SSN  in  Issue 
Due-Out  Items  Screen 


Non-Issue  Quantity 


EXIT  Due-Out  Items 
Screen _ _ 


#  times  NonDynHl p  in 
Screen _ 
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re-entry  into  Clothing  Issues 
Process  Menu  and  entry  into 
Issue  Due-Out  Items  Screen. 


Record  #  times  NonDynHlp 
key  pressed  between  re-entry 
into  Clothing  Issues  Process 
Menu  &  Complete  the  Issue  Screen. 

Record  #  times  DynHlp 
key  pressed  between  re-entry 
into  Clothing  Issues  Process 
Menu  &  Complete  the  Issue  Screen. 


2.2  Non-Issue  Quantity 
{Issue  Due-Out  Items} 

Record  #  keystrokes  made  in 
Issue  Due-Out  Items  Screen. 


Clothing  Issue  Menu 
&  Due-Out  Items  Screen 


#  times  NonDynHlp  between 

Clothing  Issues  Menu  reentry 
&  Issue  Due-Out  Items 
Screen _ _ 

#  times  DynHlp  between 
Clothing  Issues  Menu  reentry 
&  Issue  Due-Out  Items 
Screen 


Non-Issue  Quantity 


#  keystrokes  in  Issue 
Due-Out  Items  Screen 
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count  *  from  THT15B  where  user= 
&  key=DynHlp  &  scr/fld= 
IssueDueOut/*. 


#  times  DynHlp  in 
Screen 


■'OTHER  ITEMS"  TO  BE  COLLECTED  {15B} 

DESCRIPTION  FORMAT  IN  SCR 


1 . 1  Fastest  Movement  &  Correction 
{Complete  the  Issue  Screen} 

Record  date/time  of  entry  into 
the  Complete  the  Issue  Screen. 

Record  #  keystrokes  from  1st 
entry  into  LIN  U06645  and 
re-entry  into  SIZE  fid  of  N39848. 

Record  Date/time  of  exit  from 
screen  (ie  <ESC>  accepted  by 
program) . 

1.2  SIZE  {Complete  the  Issue} 

Date/time  of  entry  into  SIZE 
field  for  LIN  C07440. 


Fastest  Movement  &  Correction 

Entry  into  Complete  the 
Issue  Screen _ 

#  keystrokes  LI06645  to 
N39848  _ 

EXIT  Complete  the  Issue 
Screen _ 

SIZE 

Enter  C07440 


Date/time  of  entry  into  SIZE 
field  for  LIN  N39848. 


2. 1  Menu  Selection 
{Clothing  Issues  Menu  to  Issue 

Record  date/time  of  re-entry 
into  Clothing  Issue  Process 
Menu  from  the  Complete  the 
Issue  Screen. 

Record  #  keystrokes  between 


Enter  N39848 


Menu  Selection 
Due-Out  Items) 

Re-Enter  Clothing  Issue 
Menu 


#  keystrokes  between 
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APPENDIX  9 

OBSERVATION  WORKSHEET 
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TASK  ID 


INTERRUPT  TIME: 

RESUMPTION  TIME: 

Reason  for  InterruDt: 

DEAD-END  ENCOUNTERS: 

SubtasK  ID 

Screen/Field 

Descriotion 

SubtasK  ID 

Screen/Field 

Descriotion 

SubtasK  ID 

Screen/Field 

Descriotion 

SubtasK  ID 

Screen/Field 

Descriotion 

nomb^ no  Of  Prooram: 

Other  Observations: 


4-1 
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APPENDIX  10 

PRE-TEST  QUESTIONNAIRE 
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PRE-SORVXY 


aSER  ID 

Basic  Data: 

Last  Name  _  First  Name  _  MI 

GRADE  _  Duty  MOS  _  Unit  Phone  -  _ 

Unit  Name/Address  _ 


The  highest  level  of  education  I  have  completec  is  (Circle 
appropriate  number) ; 

1  -  Grade  School 

2  -  High  School 

3-2  years  of  college  or  Associates  Degree 
H  -  Bachelors  Degree 

5  -  Enrolled  in  Masters  Degree  Prograr. 

6  -  Masters  Degree  or  higher 


Supply  Knovledoe; 

Answer  questions  1-11  by  circling  the  number  wnicn  indicates  the  correct 
response . 


Within 

Within 

within 

Within 

Longer 

Never 

How  long  ago  has  xt  been 

Past 

Past  3 

Past 

Past  3 

Than  3 

Have 

since  you; 

Month 

Months 

Year 

Years 

Yrs  aoo 

1 . Processed  paperwork  for 
tne  Turnln  of  an  item  or 
prepared  a  DA  Form  2765-1 
for  an  item  turnin? 

1 

C 

3 

4 

5 

2. Processed  paperwork  for 
a  Request  for  Issue  or 
prepared  a  DA  Form  2765-1 
for  to  request  an  issue? 

1 

2 

3 

5 

3. Entered  supply  requisi¬ 
tions  into  an  automated 
system  ( ie  keypunching 
2765-1  data)? 

1 

2 

n 

4 

4 . Prepared  or  annotated 
a  DA  Form  3645/DA  Form 
3645-1  (Manual  Clothing 
Record) ? 

1 

4 

c 

£ 
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within 

Within 

Within 

Within 

Longer 

Never 

Past 

Past  3 

Past 

Past  3 

Than  3 

Have 

Month 

Months 

year 

Years 

Yrs  ago 

5 . Processed  paperwork 

dealing  with  Statements  cf 
Charges  or  actually 
prepared  a  DD  Form  362 
(Statement  of  Charges)? 

1 

2 

3 

j. 

c 

6 . Processed  paperwork 

dealing  with  Reports 
of  Survey  or  actually 
prepared  a  DA  Form  469? 
(Report  of  Survey)? 

1 

2 

7. Maintained  a  Document 

Register? 

1 

B.  ADProxiir.atel V  how  many  US  Army  or  Installation-level  Supply  Manaqemer.t 
Courses  have  you  completed  ;to  include  AIT,  PLL  courses,  etc)? 


1  -  None 

2  -  One  to  Two 

3  -  Three  to  Four 
<  -  Five  to  six 

5  -  Seven  or  More 


Strongly  Agree  Neutral  Disagree  Strongly 
Agree  _  _  _  Disagree 

9.  I  consider  myself  to  be 

knowledgeable  in  unit  and 
organizational  supply 

functions.  1  2  3  i  i 

10.  I  consider  myself  to  be 
knowledgeable  in  direct 

support  supply  functions.  1  2  3  ■;  3 

11.  I  consider  myself  to  be 
knowledgeable  in  direct 
support  warehousing 

operations.  12343 


2 
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rl  tj 


Answer  quesrions  12-20  by  circling  tne  number  indicating  tbe  correct 
response  or  by  writing  a  response  in  the  space  provided. 


12.  Have  you  ever  worKed  in  a  Central  Issue  Facility? 

1  -  YES 

2  -  NO  (Skip  to  Question  21) 

13.  When  did  you  last  work  at  a  Central  Issue  Facility  (Year)? 


14.  How  many  total  years  CIF  working  experience  do  you  have? 

1  -  Less  than  one  year 

2  -  One  to  less  than  two  years 

3  -  Two  to  less  than  three  years 

4  -  Greater  than  three  vears 


15.  Have  you  ever  worked  at  an  automated  CIF? 

1  -  YES 

2  -  NO  (Skip  to  Question  21) 

16.  Where  was  the  last  automated  CIF  that  you  worked  at  located? 


17.  What  year  did  you  last  work  at  an  automated  CIF? 


IE.  In  the  automated  CIF,  did  you  ever  work  at  a  computer  terminal? 

1  -  YES 

2  -  NO  (Skip  to  Question  21) 

19.  In  the  automated  CIF  what  software/hardware  did  you  work  with? 

1  -  WANG  Svstem 

2  -  TRADOC  System 

3  -  Other 

4  -  Unknown 

0.  Approximately  how  many  hours  per  week  did  you  work  at  the  CIF  ccmpu 
err.inal? 

1  -  Less  than  one 

2  -  One  to  less  than  five 

3  -  Five  to  less  than  ten 

4  -  Ten  to  less  than  fifteen 

5  -  Greater  tnan  fifteen 
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computer  Knowledge: 

Answer  questions  21-27  by  circling  the  number  indicating  the  correct 
response  or  by  writing  a  response  in  the  space  provided. 


21.  I  have  completed  the  following  number  of  computer  courses  whi.e 
enrolled  in  formal  schooling  (le  graae  school,  hign  scnooi  and  college; . 


1  -  None 

2  -  One  or  Two 

3  -  Three  or  four 

4  -  Five  or  more 


22.  I  have  completed  the  following  number  of  US  Array  or  Department  cf 
Defense  courses  in  which  I  became  somewhat  familiar  with  a  c  mputer 
software. (Approximately) 


1 

2 

3 

4 

5 

6 


None 

One 

Two 

Three 

Four 

Five  or  raore 


23.  Have  you  ever  written  a  computer  program? 


1  -  YES 

2  -  NO  (Skip  to  Question  26) 


24.  When  did  you  most  recently  write  a  computer  program  (Year)? 


25.  List  any  or  all  programming  languages  in  which  you  have  done  ccmnute 
programming,  no  matter  how  trivial. 


2  - 


219 


26.  In  your  current  job,  approximately  how  many  hours  per  week  do  you  s:t 
at  a  computer  terminal  to  work? 

1  -  None 

2  -  Less  than  one  hour 

3  -  One  to  less  than  five  hours 

4  -  Five  to  less  than  ten  hours 

5  -  Ten  to  less  than  fifteen  hours 

6  -  Greater  than  fifteen  hours 

27.  I  am  able  to  type  the  followinc  numoer  of  words  per  minute 
(approximately) .  If  you  have  no  idea  how  many  words  per  minute  you  can 
type,  then  do  not  answer  this  question. 

1  -  30  or  less 

2  -  31  to  50 

3  -  51  to  70 

4  -  71  to  90 

5  -  Greater  than  90 


Attitudes  Toward  the  Computer; 

If  you  have  never  used  a  computer,  skip  questions  28-33.  Otherwise  circle 
the  response  which  most  appropriately  describes  your  feelings  and  attitudes 
while  working  on  a  computer. 


28.  I  feel  comfortable  working 
at  a  comouter  terminal. 


Strongly  Agree  Neutral  Disagree  strongly 
Agree  _ _ Disagree 


29.  I  often  have  trouble 
reading  from  a  computer 
screen. 

30.  Whenever  I  use  a  computer 
terminal ,  my  main  concern 

is  the  avoidance  of  mistakes 
or  errors  while  performing 
a  task. 

31.  Whenever  I  use  a  computer 
terminal,  my  main  concern 
is  to  perform  the  task  as 
rapidly  as  possible. 


2-5 


220 


strongly  Agree  Neutral  Disagree  strongly 
Agree  _  _  _  Disagree 

32.  Whenever  using  a  computer, 

1  prefer  to  use  a  program 
which  allows  me  to  choose 
the  function  I  wish  to 
perform  from  several  choices 
displayed  on  the  screen  over 
a  program  where  I  would  be 
required  to  type  in  commands 
in  order  to  perform  the 

function.  1234  ^ 


33.  Whenever  you  are  confused  about  how  to  enter  a  data  item  into  a 
computer  terminal,  approximately  how  many  times  will  you  attempt  to  "guess" 
the  proper  data  entry  or  format  before  you  stop  your  work  and  locate  a 
manual  or  printout  with  which  to  look  up  the  correct  entry  or  format? 


1  -  None,  I  always  look  up  items 

if  I  am  confused.  I  do  not 
make  "guess"  entries. 

2  -  Once,  then  1  go  to  look  up 

the  correct  entry  or  format. 

3  -  Twice. 

4  -  Three  or  more  times. 


Please  return  this  survey  and  all  responses  to  the  System  Administrator. 
You  will  then  be  given  further  instructions. 


2-6 
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APPENDIX  11 

POST-TASK  QUESTIONNAIRES 
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POST-TASK  IIH  SDRVTY 


OSER  ID 


1.  If  you  used  Dynamic  HELP  in  Social  Security  Nuatier  data  entry, 
indicate  all  the  applicable  points  of  confusion  or  reasons  for 
accessinq  HELP.  (Circle  all  applicable  numbers)  If  you  did  not 
use  Dynamic  HELP  in  this  case,  skip  to  question  3. 


1  -  Format  of  the  data  entry. 

2  -  Common  data  entry  mistakes  such  as  confusino 

letter  o  with  zero. 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  the  screen. 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry . 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  move  to 

next  screen. 

9  -  To  view  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  To  clarify  the  input  expected  by  example  or 

explanation. 

12  -  Needed  clearer  description  of  a  Menu  function. 

13  -  Curiosity. 

14  -  Other  _ 


2.  How  well  did  Dynamic  HELP  assist  you  with  tne  Social  Security 
Number  data  entry? 


1  -  Dynamic  HELP  was  essential  to  the  completion 

of  the  entry.  (Prevented  a  Dead-End) 

2  -  Dynamic  HELP  was  very  helpful  (le,  saved  more 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (le,  saved  10  to  30 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unhelpful  (le,  did  not  assist 

me  or  save  me  any  time) . 


IIH  -  1 
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3.  Ii  you  used  Dynamic  HELP  in  the  DIC  data  entry,  indicate  alJ 
the  applicable  points  of  confusion  or  reasons  for  accessing  HELP 
(Circle  all  applicable  numbers)  If  you  did  not  use  Dynamic  HELP 
in  this  case,  skip  to  question  5, 


1  -  Format  of  the  data  entry. 

2  -  Common  data  entry  mistakes  such  as  confusing 

letter  o  with  zero. 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  the  screen 

5  -  How  to  finish  making  the  data  entry . 

6  -  How  to  abort  the  data  entry. 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  ana  move  to 

next  screen. 

9  -  To  view  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  To  clarify  the  input  expected  by  example  or 

explanation. 

12  -  Needed  clearer  description  of  a  Menu  function 

13  -  Curiosity. 

14  -  Other 


4.  How  well  did  Dynamic  HELP  assist  you  with  the  OIC  data  entry? 


1  -  Dynamic  HELP  was  essential  to  the  completion 

of  the  entry.  (Prevented  a  Dead-End) 

2  -  Dynamic  HELP  was  very  helpful  (le,  saved  more 

than  30  seconds  of  task  time) . 

3  -  Dynamic  KELP  was  helpful  (ie,  saved  10  to  30 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unhelpful  (ie,  did  not  assis 

me  or  save  me  any  time) . 


IIH  -  2 
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5.  If  you  used  Dynamic  HELP  in  the  GRADE  dara  entry,  indicate  a 
the  applicable  points  of  confusion  or  reasons  for  accessing  H 
(Circle  all  applicable  numbers)  If  you  did  not  use  Dynamic  H 
in  this  case,  skip  to  question 


1  -  Format  of  the  data  entry . 

2  -  Common  data  entry  mistaxes  such  as  confusinc 

letter  o  with  zero. 

;  -  Movement  to  next  data  entry  in  the  screen. 

•i  -  Movement  to  previous  data  entry  in  the  scree 

5  -  How  to  finish  making  the  oata  entry. 

6  -  How  to  abort  the  data  entry. 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  move  tc 

next  screen. 

9  -  To  view  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  Key. 

11  -  To  clarify  the  input  expected  by  example  cr 

explanation. 

12  -  Needed  clearer  descripticr.  of  a  Menu  functic 

13  -  Curiosity. 

14  -  Other  _ 


6.  How  well  did  Dynamic  HELP  assist  you  with  the  GRADE  data 
entry? 


1  -  Dynamic  HELP  was  essential  to  the  complezior. 

of  the  entry.  (Prevented  a  Dead-End; 

2  -  Dynamic  HELP  was  very  helpful  (le,  saved  mcr 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (ie,  saved  10  tc  i: 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unhelpful  (ie,  did  not  assi 

me  or  save  me  any  time) . 


IIH  -  3 
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7.  If  you  used  Dynamic  HELP  to  assist  you  in  answering  the 
question  about  whether  or  not  a  Special  Issue  was  authorized, 
indicate  all  the  applicaoie  points  oi  confusion  or  reasons  ror 
accessing  HELP.  (Circle  all  applicable  numbers)  If  you  dio  nor 
use  Dynamic  HELP  in  this  case,  skip  to  question  13. 


1 

2 

3 

•i 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 


Format  of  the  data  entry. 

Common  data  entry  mistakes  such  as  confusing 
letter  o  with  zero. 

Movement  to  next  data  entry  in  the  screen. 
Movement  to  previous  data  entry  in  the  screen 
How  to  finish  making  the  data  entry. 

How  to  abort  the  data  entry. 

How  to  get  to  previous  screen. 

How  to  process  the  data  entries  ana  move  to 
next  screen. 

To  view  a  list  to  choose  from. 

Not  clear  how  to  use  the  key . 

To  clarify  the  input  expected  by  example  or 
explanation. 

Needed  clearer  description  of  a  Menu  function 
Curiosity . 

Other  _ _ 


8.  How  well  did  Dynamic  HELP  assist  you  with  the  question  about 
Special  issue  authorization? 


1  -  Dynamic  HELP  was  essential  to  completion  of 

the  qiuestion.  (Prevented  a  Dead-End) 

2  -  Dynamic  HELP  was  very  helpful  ( ie ,  saved  more 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (le,  saved  10  to  30 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unhelpful  (ie,  did  not  ass  is 

me  or  save  me  any  time) . 


IIH  -  4 
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9.  If  you  used  Dynamic  HELP  in  completing  SIZE  entries  for 
certain  LIN  items  indicate  all  the  applicable  points  of 
confusion  or  reasons  for  accessing  HELP.  (Circle  all  applicable 
numbers)  If  you  did  not  use  Dynamic  HELP  for  any  of  the  SIZE 
entries,  skip  to  question  9. 


1  -  Format  of  the  data  entry. 

2  -  Common  data  entry  mistakes  such  as  confusing 

letter  o  with  zero. 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  the  screen. 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry. 

7  -  How  to  get  to  previous  screen. 

3  -  How  to  process  the  data  entries  and  move  to 
next  screen. 

9  -  To  view  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  To  clarify  the  input  expected  by  example  or 

explanation. 

12  -  Needed  clearer  description  of  a  Menu  function. 

13  -  Curiosity. 

!•;  -  Other  _ 


10.  How  well  did  Dynamic  HELP  assist  you  with  the  SIZE  entries 
for  certain  LIN  items? 


1 

2 


j 

4 


Dynamic  HELP  was  essential  to  the  completion 
of  some  of  the  entries.  (Prevented  Dead-Ends) 
Dynamic  HELP  was  very  helpful  (le,  saved  more 
than  30  seconds  of  task  time) . 

Dynamic  HELP  was  helpful  (le,  saved  10  to  30 
seconds  of  task  time) . 

Dynamic  HELP  was  unhelpful  (ie,  did  not  assist 
me  or  save  me  any  time) . 
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11.  If  you  used  Dynamic  HELP  to  learn  how  to  use  the  Fl  Xey  (Add 
a  LIN)  or  the  F2  )cey  (Delete  a  LIN),  indicate  all  the  apoiicacie 
points  of  confusion  or  reasons  for  accessing  HELP.  (Circle  all 
applicable  numbers)  If  you  did  not  use  Dynamic  HELP  ir  is  ror 
either  of  these  iceys,  s)cip  to  question  11. 


1  -  Format  of  the  data  entry. 

2  -  Common  data  entry  mistakes  such  as  cc 

letter  o  with  zero. 

3  -  Movement  to  next  data  entry  in  the  sc 

4  -  Movement  to  previous  data  entry  in  ti 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry . 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  r 

next  screen. 

9  -  To  view  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  To  clarify  the  input  expected  by  exar: 

explanation. 

12  -  Needed  clearer  description  of  a  Menu 

13  -  Curiosity. 

14  -  Other  _ 


12.  How  well  did  Dynamic  HELP  assist  you  with  the  Pi  key  (Add  a 
LIN)  or  the  F2  )sey  (Delete  a  LIN)? 


Dynamic  HELP  was  essential  to  the  use  cf  the 
key.  (Prevented  a  Dead-End) 

Dynamic  HELP  was  very  helpful  ( ie ,  saved  r.ore 
than  30  seconds  of  task  time) . 

Dynamic  HELP  was  helpful  (ie,  saved  13  tc  30 
seconds  of  task  time) . 

Dynamic  HELP  was  unhelpful  (ie,  did  not  assis 
me  or  save  me  any  time) . 


13.  If  you  used  Dynamic  HELP  in  the  Transferred 
indicate  all  the  applicable  points  of  confusion 
accessing  HELP.  (Circle  all  applicable  numbers) 
use  Dynamic  HELP  in  this  screen,  skip  to  End  of 


Items  Screen, 

or  reasons  for 
If  you  did  ncr 
Survey . 


1  -  Format  of  the  data  entry. 

2  -  Common  data  entry  mistakes  such  as  confusinc 

letter  o  with  zero. 

3  “  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  the  screen 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry. 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  move  to 

next  screen. 

9  -  To  view  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  To  clarify  the  input  expected  by  example  or 

explanation. 

12  -  Needed  clearer  description  of  a  Menu  function 

13  -  Curiosity. 

14  -  Other  _ 


14 .  How  well  did  Dynamic  HELP  assist  you  with  the  Transferred 
Items  Screen? 


1  -  Dynamic  HELP  was  essential  to  the  completion 

of  the  screen  entries.  (Prevented  a  Dead-End) 

2  -  Dynamic  HELP  was  very  helpful  (ie,  saved  more 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (ie,  saved  10  to  30 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unhelpful  (ie,  did  not  assis 

me  or  save  me  any  time) . 


EKD  OF  POST-TASK  IIH  SPRVEY 
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CSER  ID 


1.  During  this  task,  if  the  social  Security  Numher  data  entr>’  was 
confusing,  indicate  all  the  applicable  points  of  confusion.  If 
the  data  entry  was  not  confusing,  skip  to  cfuestion  2. 


1  -  Format  of  the  data  entry. 

2  -  Confused  letter  o  with  zero  or  otner  comnor. 

data  entry  mistakes . 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  tne  screen. 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry. 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  nove  to 

next  screen. 

9  -  Needed  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  Needed  an  example  or  explanation  of  the 

expected  input. 

12  -  Needed  clearer  description  of  a  Menu  function. 

13  -  Other  _ 


2.  During  this  task,  if  the  DIC  data  entry  was  confusing, 
indicate  all  the  applicable  points  of  confusion.  If  the  data 
entry  was  not  confusing,  skip  to  question  3. 


1  -  Format  of  the  data  entry . 

2  -  Confused  letter  o  with  zero  or  otner  common 

data  entry  mistakes. 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  m  the  screen. 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry . 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  move  to 

next  screen. 

9  -  Needed  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  Needed  an  example  or  explanation  of  tne 

expected  input. 

12  -  Needed  clearer  description  of  a  Menu  function. 

13  -  Other _ _ _ 
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3.  During  this  task,  if  the  GRADE  data  entry  was  contusing, 
indicate  all  the  applicable  points  of  confusion.  :f  rne  cata 
entry  was  not  conrusinq,  skip  to  question  . 


4 

5 

6 

8 

9 

10 


12 


Format  of  the  data  entry. 

Confused  letter  o  with  zero  or  otr.'^r 
data  entry  mistakes. 

Movement  to  next  data  entry  i.n  t.he  s 
Movement  to  previous  data  entry  in  t 
How  to  finisn  making  the  cata  entry. 
How  to  abort  the  data  entry . 

How  to  get  to  previous  screen. 

How  to  process  the  data  entries  ano 
next  screen. 

Needed  a  list  to  cnoose  frc.t . 

Not  clear  how  to  use  the  key , 

Needed  an  example  or  explanation  c: 
expected  input . 

Needed  clearer  description  of  a  Menu 
Other  _ 


conmo.n 


cree.i . 

.n  e  screen. 


tne 


f unct 1  or 


4.  During  this  task,  if  the  question  about 
soldier  was  authorized  a  Special  Issue  was 
all  the  applicable  points  of  confusion.  If 
confusing,  skip  to  question  5. 


wnetner  or  not  a 
confusing,  inoicate 
the  question  was  not 


1  -  Format  of  the  data  entry. 

2  -  Confused  letter  o  with  zero  or  ctner  conmor 

data  entry  mistakes . 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  tne  screen 

5  -  How  to  finish  making  the  cata  entry. 

6  -  How  to  abort  the  data  entry- . 

3  -  How  to  get  to  previous  screen, 
f  -  How  to  process  the  data  entries  and  rove  to 
next  screen. 

9  -  Needed  a  list  to  choose  frcr. . 

10  -  Not  clear  how  to  use  the  r.ey . 

11  -  Needed  an  examoie  or  exoianaticn  zt  the 

expected  input. 

12  -  Neeaed  clearer  aescription  cf  a  .Menu  f-unct.c'. 

13  -  Other  _  _ 
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5.  During  this  task,  if  the  SI2E  data  entries  for  sere  cf  rnc-  LZ 
items  were  confusing,  indicate  all  the  applicable  points  c; 
confusion.  If  the  SIZE  data  entries  were  not  conrusinc,  s>;ip  sc 
question  6. 


1  -  Format  of  the  data  entry. 

2  -  Confused  letter  o  with  zero  or  otner  ccr.rcn 

data  entry  mistakes. 

3  -  Movement  to  next  data  entry  in  the  scree.n. 

A  -  Movement  to  previous  data  entry  in  tne  screen 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry . 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  move  to 

next  screen. 

9  -  Needed  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  Needed  an  example  or  explanation  cf  t.ne 

expected  input. 

12  -  Needed  clearer  description  of  a  Menu  function 

13  -  Other  _ 


6.  During  this  task,  if  use  of  the  Pi  key  (Add  a  LIN)  or  F2  key 
(Delete  a  LIN)  was  confusing,  indicate  all  the  applicable  points 
of  confusion.  If  neither  of  the  keys  was  confusing,  skip  to 
question  7 . 


1  -  Format  of  the  data  entry. 

2  -  Confused  letter  o  with  zero  or  otner  common 

data  entry  mistakes. 

3  -  Movement  to  next  data  entry  in  the  screen. 

A  -  Movement  to  previous  data  entry  in  tne  screen 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  aoort  the  data  entry . 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  anc  move  tc 

next  screen. 

9  -  Needed  a  list  to  choose  fro.r, . 

10  -  Not  clear  how  to  use  the  key. 

11  -  Needed  an  example  or  explanation  cf  the 

expected  input. 

12  -  Needed  clearer  description  of  a  Menu  function 

13  -  Otner _ _ 
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7.  During  rnis  task,  if  any  entry  or  key  in  the  Transferred  Items 
Screen  was  confusing,  inaicate  ali  the  appiicacie  pci,-.rs  c: 
confusion.  If  the  Transferred  Items  Screen  was  not  ccnrusir.a, 
skip  to  End  of  Survey. 


1  -  Format  of  the  data  entry. 

2  -  Confused  letter  o  with  zero  or  other  commor. 

data  entry  mistakes. 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  the  screen. 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  aoort  the  data  entry. 

7  -  How  to  get  to  previous  screen. 

S  -  How  to  process  the  data  entries  anc  move  to 
next  screen. 

9  -  Needed  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  Key. 

11  -  Needed  an  example  or  explanation  of  tne 

expected  input. 

12  -  Neeaed  clearer  description  of  a  Menu  function. 

13  -  Other  _  _ 


EKP  or  POST-TASK  IIN  SURVEY 


IIN  -  4 


233 


post-task  12V  SDRVEY 


USER  ir 


1.  If  you  used  Dynamic  HELP  in  the  Condition  Code  data  entry, 
indicate  all  the  applicable  points  of  confusion  or  reasons  for 
accessing  HELP.  (Circle  all  applicable  numbersl  If  you  did  ncr 
use  Dynamic  HELP  for  the  Condition  Code,  skip  to  question  1. 


1  -  Format  of  the  data  entry . 

2  -  Common  data  entry  mistakes  such  as  ccnrusi.nc 

letter  o  with  zero. 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  t.ne  screen 

5  -  How  to  finisn  making  the  data  entry. 

6  -  How  to  abort  the  data  entry. 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  rove  tc 

next  screen. 

9  -  To  view  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  To  clarify  the  input  expected  by  example  or 

explanation. 

12  -  Needed  clearer  description  of  a  Menu  functicr. 

13  -  Curiosity. 

14  -  Other  _ 


2.  How  well  did  Dynamic  HELP  assist  you  with  the  Condition  Code 
data  entry? 


1  -  Dynamic  KELP  was  essential  to  the  com.pleticr. 

of  the  entry.  (Prevented  a  Dead-End, 

2  -  Dynamic  HELP  was  very  helpful  (le,  saved  mere 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (ie,  saved  10  to 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unhelpful  (le,  did  net  assis 

me  or  save  me  any  timej . 


3.  Did  you  use  Dynamic  HELP  in  the  DRMO  Turnin  screen  to 
determine  how  to  move  to  the  Condition  Code  field  (usino  the  ESC 

key)? 


1  -  YES 

2  -  NO  (Skip  to  question  t) 
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4.  How  well  did  Dynamic  HELP  assist  you  with  determinina  how  t 
move  to  the  Condition  Code  field  (by  using  the  ESC  Icey)^ 


1  -  Dynamic  HELP  was  essential  to  the  movement  to 

the  field.  (Prevented  a  Dead-End) 

2  -  Dynamic  HELP  was  very  helpful  (le,  saved  mere 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (ie,  saved  10  to  30 

seconds  of  task  time) . 

•5  -  Dynamic  HELP  was  unhelpful  (ie,  did  not  assis 
me  or  save  me  any  time) . 


5.  Did  you  use  Dynamic  HELP  in  the  Statement  of  Charges  Screen  i 
order  to  determine  how  to  make  corrections  to  previous  entries' 


1  -  YES 

2  -  NO  (Skip  to  question  ~] 


6.  How  well  did  Dynamic  HELP  assist  you  with  determining  now  to 
make  corrections  to  previous  entries? 


1  -  Dynamic  HELP  was  essential  to  the  completion 

of  the  corrections.  (Prevented  a  Dead-End) 

2  -  Dynamic  HELP  was  very  helpful  (ie,  saved  more 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (ie,  saved  10  to  30 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unhelpful  (ie,  did  not  assis 

me  or  save  me  any  time) . 


7 .  Did  you  use  Dynamic  HELP  in  the  Statement  of  Charges  Screen  t 
determine  how  to  move  to  the  Total  Price  field  (using  the  ESC 
)cey)? 


1  -  YES 

2  -  NO  (Skip  to  question  9) 
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8.  How  well  did  Dynamic  HELP  assist  you  with  determining  how  tc 
move  to  the  Total  Price  field  (by  using  the  ESC  Xey)? 


1  -  Dynamic  HELP  was  essential  to  the  movement  tc 

the  field.  (Ftevented  Dead-Ends) 

2  -  Dynamic  HELP  ^as  very  helpful  (le,  saved  more 

than  30  secor is  of  task  time) . 

3  -  Dynamic  HELP  ^as  helpful  (le,  saved  10  to  30 

seconds  of  ta  5k  time)  . 

4  -  Dynamic  HELP  vas  unhelpful  fie,  did  not  assist 

me  or  save  me  any  time) . 


9.  While  you  were  in  the  Adjustment  Transactions  Menu,  was  it 
clear  which  menu  item  to  select  in  order  to  modify  the  Report  of 
Survey? 

1  -  YES  (Skip  to  duestion  12) 

2  -  NO 


10.  What  did  you  use  to  determine  the  correct  menu  item? 


1  -  Dynamic  HELP. 

2  -  Trial  and  Error.  (If  so,  skip  to  question  12) 

3  -  Trial  and  Error  &  Dynamic  HELP. 

4  -  Canned  HELP  -  FIO  key.  (If  so,  skip  to 

question  12) 

5  -  System. Administrator . (If  so,  skip  to 

question  12) 


11.  How  well  did  Dynamic  HELP  assist  you  in  choosing  the  correct 
menu  item  to  accomplish  the  task? 


1  -  Dynamic  HELP  was  essential  to  the  correct  menu 

selection.  (Prevented  a  Dead-End) 

2  -  Dynamic  HELP  was  very  helpful  (le,  saved  more 

than  30  seconds  of  task  time) 

3  -  Dynamic  HELP  was  helpful  (ie,  saved  10  to  30 

seconds  of  task  time) 

4  -  Dynamic  HELP  was  unhelpful  (le,  did  not  assist 

me  or  save  me  any  time) 
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12.  If  you  used  Dynamic  HELP  in  the  Total  Price  data  entry 
modifying  the  Report  of  Survey,  indicate  all  the  applicaoie 
points  of  confusion  or  reasons  for  accessing  HELP.  .Circle  all 
applicable  numbers  If  you  did  not  use  Dynamic  HELP  ir.  this 
case,  skip  to  question  14. 


1  -  Format  of  the  data  entry. 

2  -  Common  data  entry  mistakes  sucn  as  ccr.iusir.a 

letter  o  with  zero. 

3  -  Movement  to  next  data  entry  in  tne  screen. 

4  -  Movement  to  previous  data  entry  in  tne  screen. 

5  -  How  to  finish  making  the  data  entr.-. 

6  -  How  to  abort  the  data  entry. 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  anc  rove  to 

next  screen. 

9  -  To  view  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  To  clarify  the  input  expected  by  example  or 

explanation. 

12  -  Needed  clearer  description  of  a  Menu  function. 

13  -  Curiosity. 

14  -  Other  _ 


13 .  How  well  did  Dynamic  HELP  assist  you  with  the  Total  Price 
data  entry  while  modifying  the  Report  of  survey? 


1  -  Dynamic  HELP  was  essential  to  the  completion 

of  the  entry.  (Prevented  a  Dead-End; 

2  -  Dynamic  HELP  was  very  helpful  (le,  savea  more 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (le,  saved  10  to  10 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unnelpful  (le,  die  net  assist 

me  or  save  me  any  time) . 


END  or  POST-TASK  12H  SURVEY 
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POST-TASK  12N  SDRVEY 


USER  ID 


1.  During  this  task,  if  the  Condition  Code  data  entry  was 
confusing,  indicate  all  the  applicable  points  of  confusion.  If 
the  data  entry  was  not  confusing,  skip  to  question  2. 


1  -  Format  of  the  data  entry. 

2  -  Confused  letter  o  with  zero  cr  otner  common 

data  entry  mistakes . 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  rne  screen. 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry. 

1  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  move  tc 

next  screen. 

9  -  Needed  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  Needed  an  example  or  explanation  of  the 

expected  input . 

12  -  Needed  clearer  description  of  a  Menu  function. 

13  -  Other  _ 


2.  While  in  the  ORMO  Tumln  screen,  was  it  clear  how  to  move  from 
the  NSN  and  QTY  section  of  the  screen  to  the  Condition  Code 
field? 


1  -  YES 

2  -  NO 


3.  In  the  Statement  of  Charges  Screen,  was  it  clear  how  tc  maKS 
corrections  to  previous  entries  in  the  NSN  and  QTY  section  cf  tne 
screen? 


1  -  YES 

2  -  NO 


4.  While  in  the  statement  of  charges  screen,  was  it  clear  now  tc 
move  to  the  Total  Price  field  from  the  NSN  and  QTY  section  cf  t.ie 
screen? 


1  -  YES 

2  -  HO 
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5.  While  you  were  in  the  Adjustment  Transactions  Menu,  li  i: 
unclear  how  to  select  the  menu  item  to  modify  the  Report  of 
Survey,  the  difficulty  was:  (If  it  was  clear,  skip  to 

question  6) 


1  -  The  meaning  of  Menu  Item  Names  was  unclear. 

2  -  The  Task  itself  was  unclear. 

3  -  Other 


6.  While  modifying  the  Report  of  Survey,  if  the  Total  Price  data 
entry  was  confusing,  indicate  all  the  applicaole  points  of 
confusion.  If  the  data  entry  was  not  confusing,  skip  to  End  of 
Survey . 


Format  of  the  data  entry. 

Confused  letter  o  with  zero  or  otner  commor 
data  entry  mistakes. 

Movement  to  next  data  entry  in  the  screen. 
Movement  to  previous  data  entry  in  the  scic 
How  to  finish  making  the  data  entry. 

How  to  abort  the  data  entry. 

How  to  get  to  previous  screen. 

How  to  process  the  data  entries  and  move  t: 
next  screen. 

Needed  a  list  to  choose  from. 

Not  clear  how  to  use  the  key. 

Needed  an  example  or  explanation  of  the 
expected  input . 

Needed  clearer  description  of  a  Menu  funct; 
Other 


END  or  POST-TASK  12N  SURVEY 


DSER  ID 


1.  If  you  used  Dynamic  HELP  while  creating  tne  first  four  Supply 
Requisitions  of  this  task,  indicate  all  the  applicable  points  of 
confusion  or  reasons  for  accessing  HELP.  (Circle  ail  appiicacie 
numbers)  If  you  did  not  use  Dynamic  HELP  for  the  first  four 
Supply  Requisitions  of  this  task,  skip  to  question  3. 

1  -  Format  of  the  data  entry. 

2  -  Common  data  entry  mistakes  such  as  confusmc 

letter  o  with  zero. 

3  -  Movement  to  next  data  entry  in  tne  screen. 

4  -  Movement  to  previous  data  entry  in  the  screen. 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry. 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  move  to 

next  screen. 

9  -  To  view  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  To  clarify  the  input  expected  by  example  or 

explanation. 

12  -  Needed  clearer  description  of  a  Menu  function. 

13  -  curiosity. 

14  -  other  _ 


2.  How  well  did  Dynamic  HELP  assist  you  with  the  first  four 
Supply  Requisitions  entries? 


1  -  Dynamic  HELP  was  essential  to  tne  completion 

of  the  requisitions.  (Prevented  a  Dead-End 

2  -  Dynamic  HELP  was  very  helpful  fie.  saved  more 

than  30  seconds  of  task  time). 

3  -  Dynamic  HELP  was  helpful  (le,  savec  1C  tc  ;c 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unhelpful  (le,  did  not  assist 

me  or  save  me  any  time) . 
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3.  If  you  used  Dynamic  HELP  while  creating  a  Supply  Requisition 
from  the  unreadable  NSN,  indicate  all  the  applicable  points  c: 
confusion  or  reasons  for  accessing  HELP.  (Circle  all  applicable 
numbers)  If  you  did  not  use  Dynamic  HELP  in  this  case,  skip  tc 
question  5. 


1  -  Format  of  the  data  entry. 

2  -  Common  data  entry  mistakes  sucn  as  confusinc 

letter  o  with  zero. 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  the  screen 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry . 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  move  tc 

next  screen. 

9  -  To  view  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  To  clarify  the  input  expected  by  example  cr 

explanation. 

12  -  Needed  clearer  description  of  a  Menu  function 

13  -  Curiosity. 

14  -  Other  _ 


4.  How  well  did  Dynamic  HELP  assist  you  with  the  creation  cf  the 
Supply  Requisition  for  the  unreadable  NSN? 


1  -  Dynamic  HELP  was  essential  to  the  completion 

of  the  requisition.  (Prevented  a  Dead-End; 

2  -  Dynamic  HELP  was  very  helpful  (le,  savec  more 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (le,  saved  1C  tc  2: 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unhelpful  (le,  did  not  ass  is 

me  or  save  me  any  time) . 


5.  While  you  were  in  the  Document  Register  Actions  Menu,  was  it 
clear  which  menu  item  to  select  in  order  to  modify  tne  quantity 
of  the  Supply  Requisition  ordered  earlier  m  the  oay? 

1  -  YES  (Skip  to  Question  8) 

2  -  NO 
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6.  What  did  you  use  to  determine  the  correct  menu  item  to  modify 
the  quantity  of  the  Supply  Requisition? 


1  -  Dynamic  HELP. 

2  -  Trial  and  Error.  (If  so,  skip  to  question  8) 

3  -  Trial  and  Error  &  Dynamic  HELP. 

4  -  Canned  HELP  -  FIO  key.  (If  so,  skip  to 

question  8) 

5  -  System  Administrator. (If  so,  skip  to 

question  8) 

7.  How  well  did  Dynamic  HELP  assist  you  in  cnoosing  the  correct 
menu  item  to  accomplish  the  task? 

1  -  Dynamic  HELP  was  essential  to  the  correct  menu 

selection.  (Prevented  a  Dead-End) 

2  -  Dynamic  HELP  was  very  helpful  (le,  saved  more 

than  30  seconds  of  task  time) 

3  -  Dynamic  HELP  was  helpful  (ie,  saved  10  to  30 

seconds  of  task  time) 

4  -  Dynamic  HELP  was  unhelpful  (ie,  did  not  assist 

me  or  save  me  any  time) 
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POST-TASK  13M  SDRVZY 


USER  ID 


1.  While  creating  the  first  four  Supply  Reqtiisitions  in  this 
task,  if  any  entries  in  the  Create  Supply  Requisitions  Screen 
were  confusing,  indicate  all  the  applicable  points  of  confusicr. . 
If  the  entries  were  not  confusing,  skip  to  question 


1  -  Format  of  tne  data  entry . 

2  -  Confused  letter  o  with  zero  or  other  comror. 

data  entry  nistakes. 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  tne  screen 

5  -  How  to  finish  making  the  aata  entry. 

6  -  How  to  abort  the  data  entry. 

7  -  How  to  aet  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  move  to 

next  screen. 

9  -  Needed  a  list  to  choose  from. 

10  -  Not  clear  .now  to  use  tne  key. 

11  -  Needed  an  example  or  explanation  cf  the 

expected  input . 

12  -  Needed  clearer  description  cf  a  Menu  functior 

13  -  Other  _ _ 


2.  While  creating  a  supply  Requisition  from  an  unreadable  NSh’  ir 
this  task,  if  any  entries  in  tne  create  Supply  Requisitions 
Screen  were  confusing,  indicate  all  the  applicable  points  cf 
confusion.  If  the  entries  were  not  confusing,  skip  to  question  . 


1  -  Format  of  the  data  entry . 

2  -  Confused  letter  o  with  zero  or  other  common 

data  entry  mistakes . 

3  -  Movement  tc  next  data  entry  in  tne  screen. 

4  -  Movement  tc  previous  aata  entry  ir.  tne  screen 

5  -  How  tc  finish  making  tne  data  entry. 

6  -  How  to  aoort  the  data  entry . 

7  -  How  to  get  tc  previous  screen. 

8  -  How  tc  process  the  data  entries  ana  move  tc 

next  screen . 

9  -  Needed  a  list  to  choose  from. 

10  -  Not  clear  now  to  use  the  key. 

11  -  Needed  an  example  or  explanation  cf  the 

expected  input. 

12  -  Needed  clearer  description  of  Menu  function. 

13  -  Other  _ 


I3N  -  1 


243 


3.  While  you  were  in  the  Dociiment  Register  Actions  Menu,  i 
was  unclear  how  to  select  the  menu  item  to  modity  rne  quan 
the  Supply  Requisition  ordered  earlier  in  tne  aav,  r.ie  d:r 
was:  (If  It  was  clear,  skip  to  question  End  ct  Survey 

1  -  The  meaning  of  Menu  Item  Names  was  unclear. 

2  -  The  Task  itself  was  unclear. 

3  -  Other _ _  _ 


END  OF  POST-TASK  13N  SURVEY 


13N  -  2 


244 


POST-TASK  ISH  SDRVEY 

USER  ID 


1.  While  you  were  in  the  Complete  the  Issue  screen,  if  you  used 
Dynamic  HELP  in  completing  SIZE  entries  for  certain  Lit.’  items, 
indicate  all  the  applicable  points  of  confusion  or  reasons  for 
accessing  HELP.  (Circle  all  applicable  numoers)  If  you  did  not 
use  Dynamic  HELP  for  any  of  the  SIZE  entries,  skip  to  question 

1  -  Format  of  the  data  entry. 

2  -  Common  data  entry  mistaKes  sucn  as  confusino 

letter  o  with  zero. 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  the  screen. 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry. 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  tne  data  entries  anc  move  to 

next  screen. 

9  -  To  view  a  list  to  choose  from. 

10  -  Not  clear  now  to  use  tne  Key. 

11  -  To  clarify  the  input  expected  by  example  or 

explanation. 

12  -  Needed  clearer  description  of  a  Menu  function. 

13  -  Curiosity. 

14  -  other  _ 


2.  How  well  did  Dynamic  HELP  assist  you  wit.n  the  SIZE  entries  for 
certain  LIN  items? 


1  -  Dynamic  KELP  was  essential  to  the  completion 

of  some  of  the  entries.  .'Prevented  Deao-Enos) 

2  -  Dynamic  HELP  was  very  nelpful  lie,  savea  more 

than  30  seconds  of  task  timej  . 

3  -  Dynamic  KELP  was  helpful  iie,  savec  10  to  lO 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unneipful  ;ie,  did  not  assist 

me  or  save  me  any  time) . 


3.  Did  you  use  Dynamic  HELP  in  the  Complete  the  issue  Screen  to 
determine  the  fastest  way  to  move  through  a  Worksheet'’ 


1  -  YES 

2  -  NO  (Skip  to  cuestior.  S) 
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•; .  How  well  did  Dynamic  HELP  assist  you  with  moving  quicKly 
through  the  Worksheet? 


1  -  Dynamic  HELP  was  essential  to  quick  movement 

through  the  Worksheet.  (Prevented  a  Dead-End) 

2  -  Dynamic  HELP  was  very  helpful  (ie,  saved  more 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (le,  saved  10  to  3C 

seconds  of  task  time). 

4  -  Dynamic  HELP  was  unhelpful  (le,  did  not  assis 

me  or  save  me  any  time) . 


5.  Did  you  use  Dynamic  HELP  in  the  Complete  the  Issue  Screen  in 
order  to  determine  how  to  make  corrections  to  previous  entries? 


1  -  YES 

2  -  NO  (Skip  to  question  ”) 


6.  How  well  did  Dynamic  HELP  assist  you  with  determining  how  to 
make  corrections  to  previous  entries? 


1  -  Dynamic  HELP  was  essential  to  the  completion 

of  the  corrections.  (Prevented  a  Dead-End) 

2  -  Dynamic  HELP  was  very  helpful  (ie,  saved  more 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (le,  saved  10  to  30 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unhelpful  (le,  did  not  assis 

me  or  save  me  any  time) . 


0.  While  you  were  ir.  the  Clothing  issues  Process  Menu,  was  if 
clear  which  menu  item  to  select  m  order  to  issue  the  due-out 
items  to  the  soldier? 

1  -  YES  (Skip  to  question  ID) 

2  -  NO 
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e.  What  did  you  use  to  determine  the  correct  menu  iterr* 


1  -  Dynamic  HELP. 

2  -  Trial  and  Error.  (If  sc,  skip  to  questic,-!  10; 

3  -  Trial  and  Error  i  Dynamic  HELP. 

4  -  Canned  HELP  -  FIO  key.  (If  so,  skit  tc 

question  10) 

5  -  System  Administrator.  (If  sc,  skip  to 

question  10) 


9.  How  well  did  Dynamic  .HELP  assist  you  in  cnoosinq  tr.e  correct 
menu  item  to  accomplisr.  the  task? 


1  -  Dynamic  HELP  was  essential  tc  t.he  correct  menu 

selection.  (Prevented  a  Dead-End) 

2  -  Dynamic  HELP  was  very  helpful  ,ife,  saved  more 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (le,  saves  1C  tc  3 C 

seconds  of  task  time) . 

4  -  Dynamic  HELP  was  unhelpful  (le,  did  net  assist 

me  or  save  me  any  time) . 


10.  While  in  the  Issue  Due-out  Items  Screes,  if  you  used  Dynamic 
HELP  in  the  DOQTY  data  entry,  indicate  all  the  applicaole  points 
of  confusion  or  reasons  for  accessing  HELP.  (Circle  all 
applicable  numbers)  If  you  did  not  use  Dynamic  KELP  in  t.his 
case,  skip  to  question  12. 


1  -  Format  of  the  data  entry. 

2  -  Common  data  entry  mistakes  suc.n  as  confusing 

letter  o  with  zero. 

3  -  Movement  to  next  data  entry  in  tne  screen. 

4  -  Movement  tc  previous  data  entry  m  tne  screen. 

5  -  How  to  finish  making  the  date  entry. 

6  -  How  to  abort  the  data  entry . 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  tne  data  entries  ana  move  tr 

next  screen. 

9  -  Tc  view  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  To  Clarify  the  input  expected  cy  example  cr 

explanation. 

12  -  Needed  clearer  description  of  a  Menu  function. 

13  -  Curiosity. 

14  -  Other  _ 
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11.  How  well  did  Dynamic  HELP  assist  you  with  the  DOQTY  data 
entry? 


1  -  Dynamic  HELP  was  essential  to  the  completion 

of  the  entry.  (Prevented  a  Dead-End) 

2  -  Dynamic  HELP  was  very  helpful  (ie,  saved  rorc- 

than  30  seconds  of  task  time) . 

3  -  Dynamic  HELP  was  helpful  (le,  saved  10  to  3C 

seconds  of  task  time) . 

4  -  Dynamic  KELP  was  unhelpful  (le,  did  not  assist 

me  or  save  me  any  time) . 
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POST-TASK  15N  SDRVTY 

OSER  ID 


1.  While  you  ■were  in  the  Complete  the  Issues  Screen,  if  tne  SIZE 
data  entries  fcr  some  of  the  LIN  items  were  confusing,  indicate 
all  the  applicable  points  of  confusion.  If  tne  SIZE  data  entries 
were  not  confusing,  skip  to  question  2. 

1  -  Format  of  the  data  entry . 

2  -  Confused  letter  o  with  zero  or  ctner  common 

data  entry  mistakes . 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  m  the  screen. 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry . 

-  How  to  get  to  previous  screen, 
t  -  How  to  process  the  oata  entries  and  move  tc 
next  screen. 

9  -  Needed  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  t.ne  Key. 

11  -  Needed  an  example  cr  e>:pa.anaticn  o:  the 

-expected  input. 

12  -  Needed  clearer  description  of  a  Menu  function. 

13  -  Other  _ _ 


2.  While  in  the  Complete  the  Issue  Screen .  was  tne  method  of 
fastest  movement  through  the  Worksheet  clear? 


1  -  YES 

2  -  NO 


3.  In  the  Complete  the  Issue  Screen,  was  :t  ziear  no.,  tc  miaKe 
corrections  to  previous  entries  or.  the  screen'? 


1  -  YES 

2  -  NO 
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4.  While  you  were  in  the  Clothing  Issue  Process  Menu,  if  it  was 
unclear  how  to  select  the  menu  item  to  Issue  Due-Out  Items  to  a 
soldier,  the  difficulty  was:  (If  it  was  clear,  sxip  to 

question  5) 


1  -  The  meaning  of  Menu  Item  Names  was  unclear. 

2  -  The  Task  itself  was  unclear. 

3  -  Other  _ 


5.  While  in  the  Issue  Oue-Out  Items  Screen,  if  the  DOQTY  data 
entry  was  confusing,  indicate  all  the  applicable  points  of 
confusion.  If  the  data  entry  was  not  confusing,  snip  to  Ena  cf 
Survey . 


1  -  Format  of  the  data  entry . 

2  -  Confused  letter  o  with  zero  or  other  common 

data  entry  mistakes. 

3  -  Movement  to  next  data  entry  in  the  screen. 

4  -  Movement  to  previous  data  entry  in  the  screen. 

5  -  How  to  finish  making  the  data  entry. 

6  -  How  to  abort  the  data  entry. 

7  -  How  to  get  to  previous  screen. 

8  -  How  to  process  the  data  entries  and  move  to 

next  screen. 

9  -  Needed  a  list  to  choose  from. 

10  -  Not  clear  how  to  use  the  key. 

11  -  Needed  an  example  or  explanation  of  the 

expected  input. 

12  -  Needed  clearer  description  of  a  Menu  function. 

13  -  Other  _ _ 


gWD  OF  POST-TASK  15N  SORVEY 
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APPENDIX  12 

POST-TEST  QUESTIONNAIRE 
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POST-TEST  SORVEY 


USER  ID 


1.  Fron  "1"  ("most  often")  to  "5"  ("least  often"),  mark  eacr.  c: 
the  following  hinds  of  Dynamic  HELP  you  sought  in  Data-Entry 
screens : 


Help  in  detemining  where  you  were  and  what 
you  could  do  there  (Location  and  Function.)  . 


Help  in  getting  to  another  screen  or  oach  to  c 
previous  screen  (Movement  between  screens). 

Help  in  getting  to  next  data  entry  or  bach  ti 
a  previous  data  entry  (Movement  -..ithin 
screen) . 

Help  with  "how  to  enter":  now  to  complete  the 
entry  in  a  field,  how  to  correct  an  entry,  r.c\. 
to  indicate  you  were  finished  with  a  data 
entry,  how  to  deal  with  a  "quirh",  how  tc 
abort  a  data  entry,  what  was  the  effect  cf 
certain  keys,  etc. 

Help  with  "what  to  enter":  meaning  of  the 
entry  in  a  field,  range  of  acceptable  entries 
in  a  field,  where  to  find  a  datum  on  a 
document  so  it  can  be  entered  on  the  screen, 
examples  cf  possible  entries,  etc. 


2.  From  "1"  ("most  often")  to  "3"  ("least  often"),  nark  eac.n  cf 
the  following  hinds  of  Dynamic  HELP  you  sougnt  in  Menu  Selection 
screens ; 


General  Orientation:  .Help  in  determining  ..■nere 
you  came  rrcm,  '..nere  you  are,  and  '..nere  you 
can  go  oach  tc. 

Help  in  understandinc  the  meaning  of  screens 
or  functions  listed. 

Help  with  procedures :  how  to  make  selections, 
how  to  aoort  a  data  entry,  what  was  tne 
effect  of  certain  keys,  etc. 
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3.  How  helpful  IS  Dynamic  HELP  in  fhe  CIF  software  v.hen  you 
compare  it  to  the  CIF  software  without  Dynamic  HELP? 

1  -  Very  Helpful 

2  -  Somewnat  Helpful 

3  -  No  Difference  in  Help 

4  -  Somewnat  Unneipfui 

5  -  Very  Unhelpful 


4.  If  possible,  Dynamic  HELP  should  be  included  in  ail  complex  'JZ 
Army  interactive  computer  proctrams  (similar  to  CIF)  . 

1  -  Strongly  Agree 

2  -  Agree 

2  -  Neutral 

4  -  Disagree 

5  -  Strongly  Disagree 


END  OF  POST-TEST  SURVEY 
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USER  TESTING  DATA 
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Ihcr  =  VJsr  =  User  Jrtcniificalion 

l  ime  =  #  of  seconds  difference 

f-ir  =  Er  =  Error  difference 

UEs  =  Dead  End  difference 

Dylllp  -=  Ull  =  «  limes  Dyn  Help  used 
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Err  =  Cr  =  Error  difference 
1)1:5  =  Head  End  difference 
Uj  Mlp  -  Dll  =  rr  limes  Dyn  Help  used 
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Dylllp  -  Dll  •  *•  limes  Dyn  Help  used 
Isi  •  seconds  until  help  called 
Ijuh  •  »  wrong  guesses  until  help  called 
iVr  «  difference  in  *  of  wrong  guesses 
llli  *  lime  lo  entry  aficr  help  called 
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Isl  •  «  seconds  unlil  help  called 
(iuh  •  M  wrong  guesses  unlil  help  called 
Wr  =  difference  in  «  of  wrong  guesses 
llli  =  lime  1(1  enliy  al  ter  help  called 
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Is)  -  «  seconds  unlil  help  called 
Guh  =  «  wrong  guesses  unlil  help  called 
Wr  »  difference  in  **  of  wrong  guesses 
Uti  =  linre  to  entry  after  helf)  called 
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(iuli  -=  •*  wrong  guesses  unlit  help  called 
Wr  =  difference  in  ft  of  wrong  guesses 
III!  =  lime  lo  eniry  aflei  help  called 
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Guh  =  *  wrong  guesses  until  help  called 
Wr  =  difference  in  «  of  wrong  guesses 
I  lli  =  lime  to  entry  at  ter  help  called 
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